Submersion Cooling Evaluation - ETCC · project, contact [email protected] or [email protected]. Legal Notice...

PG&E’s Emerging Technologies Program ET13PGE1101

Submersion Cooling Evaluation

ET Project Number: ET13PGE1101

Project Manager: Jeff Beresini Pacific Gas and Electric Company Prepared By: Luke Werner, P.E. Energy Resources Integration (ERI) 251 Donahue St. Sausalito, CA 94965

Issued: 2/25/14

Copyright, 2014, Pacific Gas and Electric Company. All rights reserved.

1


Acknowledgements

Pacific Gas and Electric Company’s Emerging Technologies Program is responsible for this project. It was developed as part of Pacific Gas and Electric Company’s Emerging Technology program under internal project number ET13PGE1101. Energy Resources Integration conducted this technology evaluation for Pacific Gas and Electric Company with overall guidance and management from Jeff Beresini and Sam Newman. For more information on this project, contact [email protected] or [email protected].

Legal Notice

This report was prepared for Pacific Gas and Electric Company for use by its employees and agents. Neither Pacific Gas and Electric Company nor any of its employees and agents:

(1) makes any written or oral warranty, expressed or implied, including, but not limited to those concerning merchantability or fitness for a particular purpose;

(2) assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, process, method, or policy contained herein; or

(3) represents that its use would not infringe any privately owned rights, including, but not limited to, patents, trademarks, or copyrights.

2


ABBREVIATIONS AND ACRONYMS

2N Type of redundancy where the components (N) have double the amount of compontents necessary for operation.

AB32 Assembly Bill 32

CFM Cubic Foot per Minute

CRAC Computer Room Air Conditioner

CRAH Computer Room Air Handler

CTE Cooling Tower Emulator

DCIM Data Center Infrastructure Management

DEER Database for Energy Efficienct Resources

Delta-T Temperature Difference

EUL Expected Useful life

FLOPS Floating Point Operations per Second

GRC Green Revolution Cooling

IT Information Technology

HP HorsePower

kW kilo-Watt

kWh kilo-Watt per hour

N+1 Type of redundancy where the components (N) have one backup compontent (+1)

MW Mega-Watt

VFD Variable Frequency Drive

3


FIGURES

Figure 1 Air Cooled Environment ........................................... 5

Figure 2 Air Cooled Environment ........................................... 5

Figure 3 Chilled Water In-Row Cooled Environment ................. 5

Figure 4 Servers Submerged ................................................ 6

Figure 5 GRC CarnotJet System ............................................ 6

Figure 6 Servers Submerged ................................................ 7

Figure 7 Motherboard Submerged ......................................... 7

Figure 8 GRC Carnot JetTM System and Cooling Tower

Emulator ............................................................. 7

Figure 9 Simple Schematic of Existing Air Cooled System ....... 11

Figure 10 CarnotJet™ system and Cooling Tower Emulator..... 18

Figure 11 CarnotJetTM Rack Loading ..................................... 20

Figure 12 Coincident Peak kW Draw .................................... 32

Figure 13 Estimated Annual Energy Consumption .................. 33

Figure 14 Theoretical Energy Consumption for RePiping Condenser Water to GRC CarnotJetTM .................... 37

Figure 15 Theoretical Energy Savings for RePiping Condenser Water to GRC CarnoJetTM ..................................... 37

Figure 16 GRC Online Systems Monitor Data ........................ 40

Figure 17 Sentilla Online Dashboard .................................... 41

TABLES

Table 1 Test Setpoints and Results ........................................ 2

Table 2 Estimated Annual Energy and Cost Savings ................. 2

Table 3 Baseline Air Management Scenarios ......................... 10

Table 4 Test Case Time and Temperature Details .................. 19

Table 5 Data Points List ..................................................... 21

Table 6 Test Results Data Summary* .................................. 22

4


Table 7 Summary of Annual Energy and Demand Consumption ..................................................... 23

Table 8 Summary of Energy and Demand Savings ................ 23

Table 9 Baseline Costs ....................................................... 24

Table 10 Technology Assessment Costs ............................... 24

Table 11 Typical Project Costs ............................................ 24

Table 12 Expected Useful Life ............................................. 25

Table 13 Test #1 Data Summary ........................................ 25

Table 14 Test #2 Data Summary* ...................................... 26

Table 15 Test #3 Data Summary* ...................................... 26






Table 21 Test Data and Regression Analysis Predicted Pump Speed ............................................................... 28

Table 22 Baseline Air Management ...................................... 29

Table 23 Test Data Summary ............................................. 31

Table 24 Summary of Annual Energy and Demand Consumption ..................................................... 31

Table 25 Estimated Energy and Cost Savings ....................... 32

Table 26 Financial Analysis ................................................. 35

Table 27 LINPACK Data Summary ....................................... 38

Table 28 Logged Data From Tests (Truncated) ...................... 39

5


CONTENTS

Abbreviations and Acronyms ................................................................................................. ii

Figures ................................................................................................................................. iii

Tables .................................................................................................................................. iii

Contents ................................................................................................................................ v

Executive Summary ........................................................................................................ 1 1

Introduction.................................................................................................................... 4 2

Market Energy Consumption and Trends .............................................................................4 2.1

Current Technology to Remove Heat ....................................................................................4 2.2

Submersion Cooling History .................................................................................................5 2.3

Submersion Cooling Technology to be Assessed ...................................................................5 2.4

Scope of Technology Assessment .........................................................................................8 2.5

Background .................................................................................................................... 9 3

Data Center Background ......................................................................................................9 3.1

Similar Technology Background ...........................................................................................9 3.2

Incumbent Technology ...................................................................................................... 10 3.3

Test Facility Technology ..................................................................................................... 11 3.4

Emerging Technology/Product ...................................................................................... 12 4

Green Revolution Cooling CarnotJetTM ............................................................................... 12 4.1

Incumbent Technologies .................................................................................................... 13 4.2

GRC CarnotJetTM Purported Advantages ............................................................................. 13 4.3

Potential market barriers .................................................................................................. 13 4.4

Assessment Objectives .................................................................................................. 15 5

Technology/Product Evaluation .................................................................................... 16 6

Evaluators Involved ........................................................................................................... 16 6.1

Evaluation Objectives ........................................................................................................ 16 6.2

Technical Approach/Test Methodology ......................................................................... 17 7

Field Testing of Technology ............................................................................................... 17 7.1

Test Plan ........................................................................................................................... 18 7.2

Instrumentation Plan ........................................................................................................ 21 7.3

6


Results .......................................................................................................................... 22 8

Summary .......................................................................................................................... 22 8.1

Incremental Cost ............................................................................................................... 24 8.2

Expected Useful Life .......................................................................................................... 25 8.3

Data Analysis .................................................................................................................... 25 8.4

Evaluations ................................................................................................................... 31 9

Cooling Performance ......................................................................................................... 31 9.1

Energy Performance .......................................................................................................... 31 9.2

Energy Savings .................................................................................................................. 32 9.3

Other Benefits ................................................................................................................... 33 9.4

Applicability to Retrofit And New Construction Program .................................................... 33 9.5

Market Barriers ................................................................................................................. 34 9.6

Areas for Future Study ....................................................................................................... 34 9.7

Financial Analysis .............................................................................................................. 35 9.8

Recommendations ........................................................................................................ 36 10

Engagement in Energy Efficiency Programs ........................................................................ 36 10.1

Suggestion for Further Investigation .................................................................................. 36 10.2

Appendices ................................................................................................................... 38 11

References .................................................................................................................... 42 12

1


EXECUTIVE SUMMARY 1

PROJECT GOAL The project goal of Submersion Cooling for Data Centers was to evaluate the technology’s ability to provide adequate cooling (heat rejection) for an existing data center’s servers and determine the technology’s energy consumption for one field installation of Submersion Cooling in a large telecommunications data center. This data was used to estimate the energy savings over theoretical baseline; with incremental project costs, it was used to estimate the simple payback period. Finally, the data was used to make recommendations for further study and potential for involvement in utility Energy Efficiency programs.

PROJECT DESCRIPTION The Submersion Cooling for Data Centers proposes to provide energy savings through submerging computational servers in a Green Revolution Cooling CarnotJetTM system. The system uses mineral oil to remove waste heat from the servers. The heated mineral oil is pumped through a heat exchanger, which is connected to the facility’s chilled water, ultimately rejecting the heat to the ambient outside air via cooling towers. Since the specific heat of the mineral oil is claimed to be approximately 1,200 times that of air by volume, the heat transfer between the servers and mineral oil is claimed to be much more effective, therefore requiring less power compared to the incumbent technology of air cooled servers.

Submersion Cooling also proposes to provide energy savings through reduced server power for a given script compared to an air-cooled environment. It is suggested this is due to a lower, and more constant, chip and processor temperature. As inclusion of this aspect is outside the scope of this study, a discussion may be found in “Areas for Further Study.”

In this retrofit installation, the servers need to be modified to be submerged in the mineral oil; servers’ fans and thermal paste need to be removed. Additionally, load banks will be installed to reach the test power density, while reducing test costs and shortening test timeline. Multiple mineral oil and cooling water temperatures will be tested to determine the technology’s ability to reject heat from the servers at both typical and extreme conditions. The cooling water temperatures will be modulated by a technology called a Cooling Tower Emulator, that according to the vendor is not typically installed, but is required to test multiple cooling water temperatures. The ability to reject heat and energy consumption required to do so will be evaluated as a part of this project.

PROJECT FINDINGS/RESULTS

The Submersion Cooling system was shown to be capable of maintaining the rack coolant temperature at setpoint for all manufacturer recommended test conditions. The system was not able to maintain the rack coolant temperature setpoint for the extreme tests, done beyond manfucturerer recommended temperatures. This was determined by observing both the the oil pump speed and tank temperatures. As the tank temperature increases, so does the oil pump speed, in order to reject additional heat to the cooling water. When the pump speed reaches 100%, the system is rejecting all the heat possible. If the temperature then continues to rise, the system is beyond its capacity.

2


During two of the tests, Test #1 and Test #4, the CarnotJetTM system was shown to approach its capacity to maintain coolant temperature. During these two tests the average recorded oil pump speed was 95% and 99%, respectively. During Test #2 and Test #3 the CarnotJetTM system was beyond its capacity, as the pump speed was at 100% and the rack coolant temperature was rising. It should be noted that the setpoints used for this test were not recommended by the manufacturer.

TABLE 1 TEST SETPOINTS AND RESULTS

Test Name

Rack Coolant Temperature Setpoint (°C / °F)

Cooling Water Temperature Setpoint (°C / °F)

Average Recorded Rack Coolant Temperature (°C / °F)

Average Recorded Cooling Water Temperature (°C / °F)

Recorded Pump Speed (%)

1 45 /113 29*/ 85 45.1 /113.2 29.0 / 85.0 95%

2 45/ 113 32 /90 46.8** /116.2 32.0 /90.0 100%

3 30 / 86 18 /65 35.1**/95.2 18.0 /65.0 100%

4 40 / 104 24 /75 40.1 / 104.2 24.0 /75.0 99%

5 45 /113 24 /75 45.1 / 113.2 24.0 /75.0 70%

6 45 /113 18 /65 45.1 / 113.2 18.0 /65.0 54%

7 40 /104 18 /65 40.2 / 104.4 18.0 /65.0 71%

8 40 /104 18 /65 40.0 / 104.0 18.0 /65.0 42%

* Product’s claimed maximum incoming cooling water temperature setpoint. **Rack coolant temperature was not maintained at setpoint.

The energy consuming equipment contained in the installed GRC CarnotJetTM system consists of two separate pumps, a GRC CarnotJetTM system Pump Module coolant pump (oil pump) and a GRC CarnotJetTM Cooling Tower Emulator pump (water pump). Over the course of the testing the electrical demand of these two pumps was measured and the average value was calculated to be 2.00 kW. As a percentage of the average server (I.T.) power the total average GRC CarnotJetTM electrical power demand was found to be 2.94%, or about 0.104 kW of pump power per ton of heat rejected.

The annual energy and peak demand, and energy cost savings (assume an annual average cost of $0.14/kWh) of the GRC CarnotJet oil pump with a CTE water pump, compared to the baseline is estimated to be:

TABLE 2 ESTIMATED ANNUAL ENERGY AND COST SAVINGS

The demand savings are 100% coincident with the utility peak demand period, as the pump and fan equipment operates at 100% duty cycle.

Demand Savings 9.25 (kW)

Energy Savings 81,030 (kWh/yr)

Energy Cost Savings 11,344$ ($/yr)

3


PROJECT RECOMMENDATIONS The assessment showed that the Green Revolution Cooling CarnotJetTM system provides for energy and peak coincident demand savings over the incumbent technology and we recommend its adoption into Energy Efficiency programs. The technology currently fits into the Customized Retrofit and Customized New Construction Programs. For an installation similar to the one installed in this project, which included a Cooling Tower Emulator system, the key variables in determining energy savings are server power, CarnotJetTM oil pump power, CarnotJetTM cooling tower emulator pump power, and the facility’s pre-existing fan power. Again the vendor stated that a Cooling Tower Emulator is not typically installed, as it ws specifically in place to for this test in order to be able to control the system cooling water temperatures with a high degree of accuracy. The technology has significant energy savings potential, a long expected useful life, and a high incremental cost. This makes it a good technology for the utility companies to Based upon conversation with the vendor there appears to be low market penetration to date, due to a variety of reasons including but not limited to the lack of off the shelf server availability, requirement to use expensive solid state drives, and uncertainty around server warranty when submerged in mineral oil. The current Database for Energy Efficiency Resources (DEER) compliance tool does not address this technology. There are still a number of suggestions that the Emerging Technologies Team suggests for further investigation, shown below.

Further Study is highly recommended for the following topics:

1) Estimate the energy and coincident demand savings potential with the cooling water coming directly from condenser water, instead of from chilled water or chilled water with a cooling tower emulator. It is hypothesized that significantly more energy and coincident demand savings will result from this anslysis.

2) Evaluate the market penetration and possible portfolio energy savings for utility energy efficiency product design and possibility of workpaper development.

3) Evaluate any energy savings claim for increased server efficiency in FLOPS per Watt. Vendor claims reduced server power and increased computation due to a lower, and more constant, chip and processor temperature. Data for this has been collected, but has need been analyzed to substantiate the claims made.

4) New Construction application – There is significant savings in Data Center Infrastructure Management through this type of installation. The avoided costs of chiller, pumps, piping, wiring, controls, building, and workers to house such reductions could be significant.

4


INTRODUCTION 2

MARKET ENERGY CONSUMPTION AND TRENDS 2.1The energy consumption and greenhouse gas emissions of data centers is of great importance as our world evolves to become more digital with cloud computing, smart phones, virtualization, and electronic-based business models. It is estimated that worldwide 1.5% of all energy consumed is by data centers (US EPA), and this will most likely continue to rise, especially with the advent of higher density servers. A typical data center will use about 100-200 times more energy per square foot than an office building (Pacific Gas & Electric Company, 2013).

Data centers are in use for the financial, banking, medical, business (online and retail), governmental, educational, telecommunications, and many other industries. These users of the data center services expect 24/7/365 instantaneous access to their data. This results in high operating costs.

The US Department of Energy estimates that the cost of power to operate a data center over its useful life is expected to exceed the cost of original capital investment (United States Deparment of Energy, 2006). Industry trends have shown recent construction of new data centers in areas with low energy costs and favorable ambient conditions to take advantage of free cooling. These trends are especially crucial to the existing data center market in California and Silicon Valley, the high-tech innovation hub, which has high energy costs and greenhouse gas limitations through legislation such as AB32 which require’s California to the state’s GHG emissions to 1990 levels by 2020. In order to maintain the existing data center population in California, and to reverse the trends of more new data centers being built elsewhere, methods to lower the operational energy cost, and therefore energy consumption, for the operation of data centers is crucial.

CURRENT TECHNOLOGY TO REMOVE HEAT 2.2The main drivers for energy consumption of a data center are the computational equipment and the removal of its by-product, heat, by mechanical equipment. Removing of heat from the computational equipment (servers) is crucial, since to operate consistently and reliably for its expected useful lifespan, they must stay below certain temperatures thresholds. These thresholds are in place for good reasons, since data centers are known as mission critical facilities, have requirements for security and reliability, and are rated by the amount of uptime.

In data centers the servers are stored in racks. In data centers with load densities of 0-10 kW/rack, the servers are typically air-cooled. As server power density increases beyond 10 kW/rack, alternative cooling strategies are required. For load densities of 10-30 kW/rack, one currently available solution is in-row cooling. This requires the piping of chilled water or refrigerant to each rack of servers. Water cooling allows for higher load densities, but adds cost, complexity, and the possibility of leaks within a data center. The figures below, from the PG&E Data Centers Best Practices document, show examples of the physical layout of these technologies.

5


FIGURE 1 AIR COOLED ENVIRONMENT

FIGURE 2 AIR COOLED ENVIRONMENT

FIGURE 3 CHILLED WATER

IN-ROW COOLED ENVIRONMENT

SUBMERSION COOLING HISTORY 2.3Submersion cooling technology is not new, as early Governmental supercomputers in the 1980s such as the Cray 2 utilized direct liquid cooling. Systems with a phase-change, liquid immersion, evaporative cooling technology such as with 3M NovecTM has also been available, though they have not been widely adopted by industry, the study of which is beyond the scope of this project. It is hypothesized that these systems have very high costs and detailed design requirements; they are not readily available “off the shelf” in the way that in-row cooling and air-cooling systems are. In fact, the PG&E Data Center Best Practices document states:

“In the future, products may become available that allow for direct liquid cooling of data center servers and equipment more directly, via methods ranging from fluid passages in chip heat sinks to submersion in a dielectric fluid. While not currently widely available, such approaches hold promise and should be evaluated as they continue to mature and are commercialized for the data center equipment market.”

SUBMERSION COOLING TECHNOLOGY TO BE ASSESSED 2.4The technology which will be assessed for this project is the Green Revolution Cooling (GRC) CarnotJetTM. The technology proposes to allow server densities between 8-40 kW/rack. The CarnotJet is a system of technologies which allows servers that have been modified to be cooled directly (called immersion or submersion) within proprietrary mineral oil, GreenDEFTM, that is thermally but not electrically conductive. The mechanical equipment is designed to have 2N redundancy, with N+1 redundancy on the heat exchangers.

6


A complete system, or module, consists of four x 42 U racks (or tanks) to make a “quad”. Each rack holds 271 gallons of GreenDEFTM mineral oil, or coolant, for a total volume of 1084 gallons. The project also required the installation of a crane. This is due to the servers being inserted from the top, instead of from the side as in a typical air cooled environment. The crane serves to also allow the servers to “drip dry” when they are removed. The CarnotJet system oil is pumped through a continuous circuit of tanks, pipes, pumps, and a heat exchanger. In this technology assessment, replaces air as the cooling medium that is directly in contact with the servers. Since the specific heat of the mineral oil is claimed to be approximately 1,200 times that of air by volume, the heat transfer between the servers and mineral oil is claimed to be much more effective, therefore requiring less power to maintain server temperatures compared with an air-cooled environment. Figures 2.4 through 2.8 show images of the CarnotJet system, servers and motherboard submerged in GreenDEFTM cooling. Figure 2.9 shows a schematic. Images are from the GRC website.

FIGURE 4 SERVERS SUBMERGED

FIGURE 5 GRC CARNOTJET SYSTEM

7


FIGURE 6 SERVERS SUBMERGED

FIGURE 7 MOTHERBOARD SUBMERGED

FIGURE 8 GRC CARNOT JETTM

SYSTEM AND COOLING TOWER EMULATOR

8


SCOPE OF TECHNOLOGY ASSESSMENT 2.5The pre-existing server cooling system at the test location is air-cooling. The air-management scheme is Level II, hot and cold isle containment with ducted-return for a Large Data Center, as described by the PG&E Energy Efficiency Baselines for Data Centers document. It consists of chilled water CRAHs, Water cooled chillers in the facility’s basement provide chilled water for the data center. Cooling tower water (condenser water) is delivered to the chillers, and is not directly available in the data center. The system to be evaluated consists of the standard CarnotJet™ equipment plus a Cooling Tower Emulator (CTE). A standard installation of a CarnotJet™ system consists of 4 x 42U racks and one pump module. The pump module provides circulation of the coolant (oil) within the server racks and is responsible for rejecting all server heat to the data center’s water source; in this case chilled water. In this specific technology assessment, the CTE is designed to be supplied with chilled water and, by means of control valves and a water pump, to deliver warmer water to the CarnotJet™ pump module. The CTE provides a means for testing the CarnotJet™ system’s capabilities at a variety of cooling water supply temperatures. Using the CTE the team was able to test at cooling water temperatures which would typically be available directly from cooling towers. The technology assessment will evaluate:

1) effectiveness at providing adequate cooling to the servers at the test load density of approximately 17.5 kW/rack, over a range of cooling water temperatures

2) energy consumption of the GRC CarnotJet™ system components’ 3) estimated energy savings from the most appropriate baseline air-flow management scheme 4) applicability to retrofit and new construction opportunities

Due to resource limitations such as measurement equipment, access to motor control panels, assessment budget, and facility personnel, this technology assessment will not:

1) Measure the energy consumption of pre-existing equipment (incumbent technology) 2) Separate the energy consumption of pre-existing facility equipment such as central plant

equipment (chiller, pumps, cooling towers), CRAHs, humidifiers, pumps, etc. 3) Evaluate any energy savings claim for reduced server power due to a lower, and more constant,

chip and processor temperature, via LINPACK test. 4) Address any hypothetical other scenarios, such as green-field new construction or brown-field

with condensing water instead of chilled water available. Though it is hypothesized that significant energy savings and Data Center Intrastructure Management gains can be made through this type of installation.

Some of these issues will be address in the suggestions for areas of further study.

9


BACKGROUND 3

DATA CENTER BACKGROUND 3.1Data centers are in use for the financial, banking, medical, retail (online and brick/mortar), oil & gas exploration, governmental, educational, telecommunications, and many other industries. These users of the data center services expect 24/7/365 instantaneous access to their data. The critical nature and 24/7/365 operation of data centers comes at a great cost. The mechanical equipment to condition the environment, and the infrastructure to house the computing equipment (server) and mechanical equiment it is expensive to design, install, maintain, and operate. The mechanical equipment typically includes include power supplies for the servers with redundancy, backup power equipment, and HVAC equipment for cooling with redundancy. The operation and management of data centers has gotten so complicated that its experts have coined a new term, Data Center Infrastructure Management (DCIM), which “is a category of solutions which were created to extend the traditional data center management function to include all of the physical assets and resources found in the Facilities and IT domains. DCIM deployments over time will integrate information technology (IT) and facility management disciplines to centralize monitoring, management and intelligent capacity planning of a data center's critical systems” (Wikipedia). This can be interpreted to meas that the management and operation of data centers and the equipment on site, has gotten to be so complex, intertwined, and expensive that efforts are taken to manage the physical assets surrounding the server.

SIMILAR TECHNOLOGY BACKGROUND 3.2Submersion cooling technology is not new, and has been used by many specialized industries for many years. Governments, universities, computer chip manufacturers’, and even utilities use similar principals to provide cooling to electrical components. However, it has not been available “off the shelf” for use as server cooling in the way the GRC CarnotJet™ technology proposes.

Early Governmental supercomputers in the 1980s, such as the Cray 2, utilized direct liquid cooling. High powered computer chip manufacturers have used similar technologies for testing purposes, prior to final module assembly (Tulkoff, 2013).

Systems with a phase-change, liquid immersion, evaporative cooling technology such as 3M NovecTM also are available, though they have not been widely adopted by the data center industry for reasons beyond the scope of this work. It is hypothesized that these systems have very high costs and detailed design requirements; they are not readily available “off the shelf” in the way that the incumbent technologies of in-row cooling and air-cooling systems are. An investigation of this technology by Lawrence Berkeley National Laboratory is being done for this technology.

Finally, a similar principle, utilizing thermally but, not electrically, conductive liquids is utilized in power distribution in transformers, which change the voltage of electrical power. However heat sinks instead of mechanical equipment is used to reject the heat to the atmospheric conditions.

10


INCUMBENT TECHNOLOGY 3.3The prevalent types of technology used in the cooling is dependent upon the IT equipment load in the data center, kW/rack, and site restrictions, among other factors (Pacific Gas & Electric Company, 2013). They are:

1. Hot Aisle/Cold Aisle, Open

2. Hot Aisle/Cold Aisle, Ducted Return

3. Hot Aisle/Cold Aisle, Fully Enclosed

4. In-Row Cooling

Details surrounding them are shown in Table 3 Baseline Air Management Scenarios

TABLE 3 BASELINE AIR MANAGEMENT SCENARIOS

The mechanical equipment used to provide heat rejection is primarily dependent upon the size of the IT load, as described within the PG&E Baseline document: Small (<1 MW IT load) Data Centers – Schemes I, II, and III Recirculation is provided by air-cooled DX computer room air conditioner (CRAC) units equipped with constant-speed fans. Specifications for CRACs from prominent manufacturers were evaluated to determine the nominal airflow per unit at an external static pressure drop of 0.3, 0.6, and 1.1 in. for air management scheme I, II, and III, respectively. Large (>1 MW IT load) Data Centers – Schemes I, II, and III Recirculation is provided by chilled water CRAHs equipped with constant-speed fans. Specifications for CRAHs from prominent manufacturers were evaluated to determine the nominal airflow per unit at the baseline external static pressure drop of 0.3, 0.6, and 1.1 in. for air management scheme I, II, and III, respectively.

11


High Density Data Centers – Scheme IV The baseline system for scheme IV is an in-row cooling solution. An in-row cooling solution is defined as a system which cools one rack or one aisle of equipment only and is physically located in the row. An in-row solution requires running chilled water or refrigerant to each rack or aisle

TEST FACILITY TECHNOLOGY 3.4

The test facility, which is a Large Data Center by the above definition, utilizes Level II air-management scheme with hot and cold isle containment and ducted-return. Air recirculation somewhat aligns with the baseline document, as it utilizes chilled water CRAHs, however it has variable speed fans. In the basement, there are water cooled chillers and chilled water circulation pumps to provide chilled water for the data center. In the basement there are also condenser water pumps to circulate condenser water to the cooling tower, which is located outside the facility. The existing servers utilize air-cooling, with four heat transfer mediums:

1) Ambient air to condenser water 2) Condenser water to refrigerant 3) Refrigerant to chilled water

4) Chilled water to data center air forced over the server

FIGURE 9 SIMPLE SCHEMATIC OF EXISTING AIR COOLED SYSTEM

Due to customer’s desired test parameters, schedule, and on-site physical piping restrictions the cooling water for the GRC system will be plumbed into the existing chilled water system, and the water temperature to the CarnotJet™’s heat exchanger will be modulated with a Cooling Tower Emulator (CTE). Thus there will still be four heat transfer mediums:

1) Ambient air to condenser water 2) Condenser water to refrigerant 3) Refrigerant to chilled water 4) Chilled water to oil, in which the server is submerged

The baseline comparison will be for a Large Data Center, Level II air-management scheme with hot and cold isle containment and ducted-return with constant speed fans. The baseline energy consumption will be from the PG&E Energy Efficiency Baselines for Data Centers document, taken with a load factor of 1.0. This will be done in order to assure that the energy savings estimates are conservative.

12


EMERGING TECHNOLOGY/PRODUCT 4

GREEN REVOLUTION COOLING CARNOTJETTM 4.1The Green Revolution Cooling (GRC) CarnotJetTM proposes to allow server densities between 8-40 kW/rack. The CarnotJet™ is a system of technologies which allows servers that have been modified to be cooled directly (called immersion or submersion) within oil that is thermally, but not electrically conductive. The mechanical equipment is designed to have 2N redundancy, with N+1 redundancy on the heat exchangers. A complete system, or module, consists of four x 42 U racks (or tanks) to make a “quad”. A system covers an area of approximately 128 square feet, with a footprint of approximately 8 feet by 16 feet. Each rack holds 271 gallons of GreenDEFTM mineral oil, or coolant, for a total volume of 1,084 gallons. It is required that the floor be level because there is a liquid involved. The system requires the installation of a few additional items that the incumbent technology does not, liquid containment and a crane. According to the manufacturer and International Building Code, secondary containment is required. The capacity required is 10% over the largest single container volume in the installation. The project also required the installation of a crane. This is due to the servers being inserted from the top, instead of from the side as in a typical air cooled environment. The crane serves to also allow the servers to “drip dry” when they are removed. The crane can be constructed on site, and made to be mobile so that it can be moved above each system as needed. In this specific technology assessment, the CTE is designed to be supplied with chilled water and, by means of control valves and a water pump, to deliver warmer water to the CarnotJet™ pump module. The CTE provides a means for testing the CarnotJet™ system’s capabilities at a variety of cooling water supply temperatures. Using the CTE the team was able to test at cooling water temperatures which would typically be available directly from cooling towers. According to GRC engineering staff the CTE is designed to mimic the operation of a cooling tower and it is their estimation that the power consumed of this equipment (a water pump) should be equivalent to the energy used by a typical cooling tower. The CarnotJet™ system oil is pumped, and in this technology assessment, replaces air as the cooling medium that is directly in contact with the servers. Since the specific heat of the mineral oil is claimed to be approximately 1,200 times that of air by volume, the heat transfer between the servers and mineral oil is claimed to be much more effective, therefore requiring less power compared to an air-cooled environment. The energy consuming equipmet consists of a pump module:

one primary 2.9 kW oil pump that is controlled by a VFD

one secondary 3.0 kW oil pump that is constant speed (backup)

and a Cooling Tower Emulator:

one primary 5.0 hp water pumps that is controlled by a VFD

one secondary 5.0 hp water pump that is constant speed (backup)

13


There are numerous sensors performing measurements throughout the system including:

Oil Pressure

Estimated filter life

Pump Command

Water temperature (multiple locations)

Oil temperature(multiple locations)

Rack temperature (multiple locations)

Power consumed

Coolant level at rack

Leak detection in pump module

Estimated heat rejection

System Health & Diagnostic Output

Based on information from GRC the CarnotJetTM system maintains the coolant temperature in the racks at a predetermined constant setpoint. While this setpoint is a “constant” in the control sequence it can be adjusted locally at the system’s controller. The typical temperature setpoint is 40°C (104°F) but at setpoints as high as 45°C (113°F) or greater are expected to be possible.

There are eight (8) rack coolant temperature sensors used in the CarnotJetTM system’s primary temperature control sequence, two in each rack. Rack coolant temperature is maintained at setpoint by continuously monitoring these eight sensors and varying the flow rate of coolant delivered to the pump module’s heat exchanger such that the maximum rack coolant temperature is kept at setpoint. Coolant flow rate is varied using a variable frequency drive housed within the pump module’s housing. An equal flow rate of coolant is delivered to each of the four racks in the typical CarnotJetTM system and there is no provision to reduce or increase the flow to any particular rack automatically.

Finally both the incumbent and technology assessed technologies are electric driven, and do not involve any natural gas. Therefore this technology assessment has no component of fuel switching.

INCUMBENT TECHNOLOGIES 4.2In data centers with load densities of 0-10 kW/rack, the servers are typically air-cooled. As server power density increases beyond 10 kW/rack, alternative cooling strategies are required. For load densities of 10-30 kW/rack, one currently available solution is in-row cooling. This requires the piping of chilled water or refrigerant to each rack of servers. This allows for higher load densities, but adds cost, complexity, and the possibility of leaks within a data center.

The equipment used for air-management schemes I-III is typically:

1. Raised floor environment of with CRAC, CRAH, or air handlers

2. Chillers & Chilled water pumps or Direct Expansion Cooling

3. Humidifiers

4. Cooling Towers and Condenser water pumps

5. Hot Aisle/Cold Aisle curtains, enclosures, ducting

13


The equipment used for air-management schemes IV, In-Row Cooling, is typically:

1. Raised floor environment of with Air Handlers

2. In-Row Cooling pumps or refrigerant compressors

3. In-Row Cooling piping and heat sinks

4. Chillers & Chilled water pumps

5. Humidifiers

6. Cooling Towers and Condenser water pumps

GRC CARNOTJETTM PURPORTED ADVANTAGES 4.3According to company literature the GRC CarnotJet™ liquid submersion cooling system for data center servers promises to:

Reduce data center cooling power by up to 95%

Reduce data center build-out costs by up to 60%

Reduce total data center power by up to 50% ongoing

Allow for rack densities of up to 40 kW/rack (for 42U rack) Additionally, in speaking with GRC representatives, they discussed other advantages over the incumbent technologies

1. Increased Server Computations (Productivity)

2. DCIM for systems that would utilize condenser water directly as the cooling water, since a chiller, chilled water pumps, controls, maintenance, and associated service contacts would not be required.

POTENTIAL MARKET BARRIERS 4.4Based upon conversations with the vendor and facility personnel, the following market barriers were identified:

1. Requires the use of solid state drives, which are expensive

a. typical magnetic style hard drives are not hermetically sealed, and allow mineral oil through their seals and ruins drive

2. Server Manufacturers do not yet offer “off the shelf” servers to submerge into mineral oil. Currently each server requires (at minimum):

a. Removal of thermal paste (clouds oil & could plug orifices)

b. Removal of air-cooling fan

3. Obtaining Server Manufacturer support to fully honor the server warranty for those modified and submerged. Currently the vendor is working with numerous server manufacturers’ to address this issue, and the server manufactuers are aware of the potential interest by customers.

4. Network gear manufacturers (switches, network interface cards, etc) lagging further behind server manufacturer

14


5. Meeting criteria for return on investment

6. Site plumbing specifics

a. Availability of condenser water within the data center is preferred

b. If this is not available, chilled water will suffice

It was asked if maintenance issues of the CarnotJet™ or servers by facility personnel was a barrier. Real Estate Facility personnel stated that they were capable of maintaining the technology assessed. IT personnel stated that the servers require extra time to remove and maintain, as the oil is dripped from the server, and it needs to be packaged in a plastic bag if it is transported. This was viewed as an invonvience, but not a barrier.

15


ASSESSMENT OBJECTIVES 5This project is a technology assessment, where the main objectives of this project are as follows:

1) Evaluate the technology’s effectiveness at providing adequate cooling to servers, at the test load density of approximately 17 kW/rack, over a range of cooling water temperatures

2) Evaluate the technology’s system components’ energy consumption

3) Determine the estimated energy savings from the most appropriate baseline air-flow management scheme

4) Applicability to retrofit and NC program

Additional Objective of the project are to:

5) Identify market barriers

6) Identify areas for further study such as

a. Green-field data centers

b. Server performance at a consistent temperature (flops/watt)

c. Server performance at varying mineral coolant temperatures

d. How to overcome market barriers

e. How to estimate energy savings compared to a baseline technology

16


TECHNOLOGY/PRODUCT EVALUATION 6 EVALUATORS INVOLVED 6.1

Luke Werner, P.E. is a principal engineer with Energy Resources Integration (ERI). ERI is an idependent firm providing services to integrate the numerous energy resources available, to both improve customers’ bottom lines, and the planet which we all call home. ERI accomplishes this through integrating their knowledge of the physical sciences and business strategies with an inquisitive nature and desire for reliable, cost effective solutions for customers. As the principal engineer, he leads ERI’s technical group in energy engineering and technology investigation. He has performed hundred of energy analysis for a wide variety of end-use customers, including the high-tech market segment of data centers. Adam Fernandez, P.E. is a senior mechanical engineer with PG&E’s Applied Technology Services (ATS) department. In his role with ATS he provides engineering and testing support to a variety of internal PG&E departments including the Customer Energy Solutions Department, the Corporate Real Estate Department, and the Power Generation Department. He is the lead data center energy efficiency engineer within ATS and has provided energy analyses and analyses review of data center projects since 2008.

EVALUATION OBJECTIVES 6.2According to company literature the GRC CarnotJet™ liquid submersion cooling system for data center servers promises to:

Reduce data center cooling power by up to 95%

Reduce data center build-out costs by up to 60%

Reduce total data center power by up to 50% ongoing

The objectives of this evaluation are to assess the performance of the GRC CarnotJet™ system related to cooling effectiveness, energy consumption, operating sequences, and to develop insight into methodology for future retrofit savings estimates. Performance data will be monitored on an installed, operating, GRC CarnotJet™ system. Criteria for the selection of a suitable installation site include:

The installation site must have cooling water available within the data center. The CarnotJet™ system rejects server heat by use of oil to water heat exchanger. Cooling water of sufficient flow rate must be available at the GRC system. Condenser water is preffered but chilled water is acceptable if the system installation includes a GRC Cooling Tower Emulator system.

The test system must include the installation of servers (and/or load banks) of sufficient power to create appreciable temperature difference between supply and return oil. The GRC system oil pump will likely have a minimum speed setpoint. An installation of insufficient load may cause the temperature difference between oil supplied to and returned from the rack to be too low to measure accurately. A 50% load factor is considered the minimum.

The selected site must allow for power monitoring using temporarily installated power meters. We anticipate installing power monitoring data loggers for all loads in the GRC system including: Server power, GRC oil pump power, GRC Cooling Tower Emulator pump power (if so equipped). The site must allow for the installation of these meters for the duration of the testing.

17


With the exception of the materials used to modify servers for oil submersion the technologies employed by GRC and used in the CarnotJet™ system are typical mechanical devices, pumps, valves, and heat exchangers.

TECHNICAL APPROACH/TEST METHODOLOGY 7The desire in evaluating the effectiveness of the CarnotJetTM system with respect to cooling capacity is to have the system loaded to as a high percentage of its capacity as the customer is agreeable to. This approach will allow the test methodology to determine the limits of the system.

Note: Achieving a high load factor is important and potentially difficult as the specified capacity of the GRC CarnotJetTM is 90 kW of server load for the 4 x 42U racks (25 kW per rack). Comparing this with a 42U server rack in an air-cooled environment (typical), which would not be loaded with more than 2.4 kW of server power, a GRC CarnotJetTM system is capable of more than 9 times typical equipment loading. In the test a server load of approximately 70 kW was accomplished.

The criteria for site selection related to evaluation, are described in Section 3.4. The only other criteria are in regards to data logger accessibility and loading, and are shown in Section 7.3.

Test cases will be run at various combinations of server loading, rack temperature setpoint, and cooling water temperature setpoint. Data will be recorded using temporary power meters, GRC internal controls data, third-party software. Each combination of server loading, rack temperature setpoint and cooling water temperature setpoint test case will be chosen to stress both the CarnotJetTM system and the installed servers and will be based on requirements of the customer.

We will use internal server temperature data, rack oil temperature data, and oil pump power data to determine the system’s capability to provide cooling during the tests. And to evaluate the electrical energy performance of the CarnotJetTM system data of server power, rack oil temperatures, Cooling Tower Emulator and Pump Module power will be recorded.

FIELD TESTING OF TECHNOLOGY 7.1The pre-existing cooling system at the test location consists of chilled water computer room air handlers (CRAHs) with partial hot and cold isle containment. In this arrangement cold air from the CRAHs is delivered to the front of the server racks then drawn through and exhausted out the back of the servers into a return air plenum. The air is continuously re-circulated, being cooled at the CRAHs with chilled water cooling coils and heated by the servers. The chilled water cooling coils within the CRAHs are served by water-cooled chillers located in the facility’s basement. Condenser water from cooling towers is delivered to the chillers and is not directly available in the data center.

A standard installation of a CarnotJetTM system consists of 4 x 42U racks and one pump module. The pump module provides circulation of the coolant (oil) within the racks and is responsible for rejecting all server heat to the data center’s water source.

The actual system in this evaluation consists of the standard CarnotJetTM equipment plus a Cooling Tower Emulator (CTE). This CTE is designed to be supplied with chilled water and, by means of control

18


valves and a water pump, to deliver warmer water to the CarnotJetTM pump module. The CTE provides a means for testing the CarnotJetTM system’s capabilities at a variety of cooling water supply temperatures. Using the CTE the team was able to test at cooling water temperatures which would typically be available directly from cooling towers. According to GRC engineering staff the CTE is designed to mimic the operation of a cooling tower and it is their estimation that the power consumed of this equipment (a water pump) should be equivalent to the energy used by a typical cooling tower.

FIGURE 10 CARNOTJET™ SYSTEM AND COOLING TOWER EMULATOR

Based on information from GRC the CarnotJetTM system maintains the coolant temperature in the racks at a predetermined constant setpoint. While this setpoint is a “constant” in the control sequence it can be adjusted locally at the system’s controller. The typical suggested temperature setpoint is between 35-40°C (95-104°F) but setpoints as high as 45°C (113°F) or greater are expected to be possible.

There are eight (8) rack coolant temperature sensors used in the CarnotJetTM system’s primary temperature control sequence, two in each rack. Rack coolant temperature is maintained at setpoint by continuously monitoring these eight sensors and varying the flow rate of coolant delivered to the pump module’s heat exchanger such that the maximum rack coolant temperature is kept at setpoint. Coolant flow rate is varied using a variable frequency drive housed within the pump module’s housing. An equal flow rate of coolant is delivered to each of the four racks in the typical CarnotJetTM system and there is no provision to reduce or increase the flow to any particular rack automatically.

To evaluate the CarnotJetTM system’s capability to maintain cooling rack coolant temperature data will be recorded at analyzed to verify temperature stability at the various test points. Additionally, Pump Module power data and control signal data will be used to shed light on pump module sequence of operations.

TEST PLAN 7.2The energy consuming components of the typical GRC CarnotJetTM system includes two water pumps (primary and secondary) which are housed within the Pump Module. For the installation under test the

19


customer also installed a Cooling Tower Emulator which also has a pump. (The control unit draws minimal power and is not considered to be a significant load.)

Installing server loads into the CarnotJetTM system is the responsibility of the customer. At the time of the testing the system was loaded to approximately 69.8 kW, which was as high a load as could be attained within the time constraints of this evaluation. Achieving this maximum load was accomplished with the use of both servers and load banks.

Operationally the installed servers ran a combination of production software and test scripts, the details of this were based on customer requirements. This was done to ensure that the consistent computational load for each test case, reducing the variables in the test data and strengthening the results. The load banks were set to draw constant power. Details of the server internal operations are not included in this report, however the Figure 11 CarnotJetTM Rack Loading below shows a diagram of the loading of the racks.

To test the system a series of test cases would be run at various combinations of server loading, rack temperature setpoint, and cooling water temperature setpoint.

Table 4 Test Case Time and Temperature Details shows the details of the setpoints and loads associated with the eight test cases.

TABLE 4 TEST CASE TIME AND TEMPERATURE DETAILS

Test Name Test Start Day

Approximate Test Start Time

Rack Coolant Temp Setpoint* °C (°F)

Cooling Water Temp Setpoint °C (°F)

Average Total Load kW

1 4-Oct 15:00 45 (113) 29 (85) 69.81

2 5-Oct 11:00 45 (113) 32 (90) 69.80

3 5-Oct 15:40 30 (86) 18 (65) 69.77

4 6-Oct 15:15 40 (104) 24 (75) 69.78

5 8-Oct 20:30 45 (113) 24 (75) 69.82

6 9-Oct 15:00 45 (113) 18 (65) 69.84

7 10-Oct 20:22 40 (104) 18 (65) 69.84

8 18-Oct 15:00 40 (104) 18 (65) 44.83

Data were recorded using temporary power meters, GRC internal controls data, and third-party software (for internal server performance data) . GRC internal controls data was chosen for the task of recording coolant and CTE water temperatures because of logistical complexities at the test site.

For each test case GRC staff programmed the CarnotJetTM coolant temperature and cooling water temperature setpoint locally at the system controller. Each test was run for a period of at least 2 hours, which was proven long enough to reach rack temperature stability. Data was collected throughout the testing and made available in spreadsheet format at the end of the testing.

20


FIGURE 11 CARNOTJETTM

RACK LOADING

GRC TANK 1 GRC TANK 3 GRC TANK 2 GRC TANK 4

U42 U42 U42 U42

U41 U41 U41 U41

U40 U40 U40 U40

U39 U39 U39 U39

U38 U38 U38 U38

U37 U37 U37 U37

U36 U36 U36 U36

U35 U35 U35 U35

U34 U34 U34 U34

U33 U33 U33 U33

U32 Open U32 U32 Open U32


U30 1.74 kW Load Bank U30 Open U30 1.74 kW Load Bank U30

U29 Open U29 Open U29 Open U29



U26 1.74 kW Load Bank U26 U26 1.74 kW Load Bank U26


U24 U24 U24 U24

U23 U23 U23 U23













U10 U10 U10 U10

U9 U9 U9 U9

U8 U8 U8 U8

U7 U7 U7 U7

U6 U6 U6 U6

U5 U5 U5 U5

U4 U4 U4 U4

U3 U3 U3 U3

U2 U2 U2 U2

U1 U1 U1 U1

APC 5.75 kW Load

Bank

Dell R910 113798

HLTD430 SN

7CR0TW1 (~500 W) HP DL380 (~200 W)

Dell R910 113798

HLTD429 SN

7CQ1TW1 (~500 W)

HP DL380 (~200 W)

3.4 kW Load Bank HP DL380 (~200 W)

4.25 kW Load Bank 4.25 kW Load Bank

APC 5.75 kW Load

Bank

APC 5.75 kW Load

Bank

APC 5.75 kW Load

Bank

APC 5.75 kW Load

Bank

3.4 kW Load Bank

Dell R910 113798

HLTD428 SN

7CR1TW1 (~500 W)

IBM 3650 (~100 W)

IBM 3650 (~100 W)

Dell R910 113798

HLTD427 SN

7CQ0TW1 (~500 W)

Dell R910 113798

HLTD432 SN

7CPZSW1 (~500W)

Dell R910 113798

HLDD102 SN

7CQZSW1 (~500 W)

Dell R910 113798

HLTD431 SN

7CQ2TW1 (~500 W)

Dell M1000 Blade

Chassis 113959

CAHYWR1VB508

SN#F1TSBX1 (~3.2

kW)

APC 5.75 kW Load

Bank

Dell M1000 Blade

Chassis 113959

CAHYWR1VB509

SN#90TSBX1 (~3.2

kW)

21


INSTRUMENTATION PLAN 7.3Input power to the servers and system oil and water pumps was monitored and recorded using a combination of seven (7) Dent Instruments Elite-Pro power meters and one Powersight PS3000 power meter. The power meters were connected at the circuit level within the data center’s cabinet power distribution units (CDUs).

Servers were powered through a combination of single-phase and three-phase power distribution units (PDUs) wired to the data center’s CDUs. Each PDU was used to power multiple servers and/or load banks and therefore direct measurements of individual server and load bank input power was not possible.

There are eight (8) rack coolant temperature sensors used in the CarnotJetTM system’s primary temperature control sequence, two in each rack. The system’s controller continuously monitors these eight sensors and maintains the worst case (warmest) sensor by adjusting coolant flow rate. The oil coolant temperature data for this testing was taken from the GRC internal controls system. CPU temperature data was made available and reported using Sentilla software. All data was recorded as 5 minute averages. Details of the monitored data points are shown in the Table 5 Data Points List.

TABLE 5 DATA POINTS LIST

Variable

Data Loggers

GRC System Data

Sentilla Data

LINPACK Data

Server Power X

Individual CDU (power strip) Power

X

Pump Module Primary Pump Power

X

Pump Module Secondary Pump Power

X

Cooling Tower Emulator Primary Pump Power

X

Cooling Water Flow X

Rack Return Coolant Temperatures (8 total, x2 per rack)

X

Rack Oil Temperature Setpoint X

Pump Module Coolant Pump Speed Signal

X

CTE Water Pump Speed Signal X

Server CPU Temperatures

X

CPU Performance X

22


RESULTS 8

SUMMARY 8.1

8.1.1 TECHNOLOGY’S EFFECTIVENESS AT PROVIDING ADEQUATE COOLING

The GRC CarnotJetTM Submersion Cooling system was shown to be capable of maintaining the rack coolant temperature at setpoint for all manufacturer recommended test conditions. The system was not able to maintain the rack coolant temperature setpoint for the extreme tests, done beyond manfucturerer recommended temperatures. Additionally the system pump energy consumption remained low relative to the server heat rejected throughout the tests. Coolant (oil) pump power and cooling tower emulator (water) pump power were, at their maximums, approximately 1.3% and 3.5% of rejected server heat respectively.

During two of the tests, Test #1 and Test #4, the CarnotJetTM system was shown to approach its capacity to maintain coolant temperature. During these two tests the average recorded oil pump speed was 95% and 99%, respectively. Since the CarnotJetTM system relies on varying oil pump speed to maintain rack coolant temperature, if the pump speed reaches 100% the system can no longer be considered to be controlling rack coolant temperature, and is beyond its capacity.

During two of the tests, Test #2 and Test #3. the CarnotJetTM system was shown to be beyond its capacity to maintain coolant temperature This was determined by observing both the the oil pump speed and tank temperatures. As the tank temperature increases, so does the oil pump speed, in order to reject additional heat to the cooling water. When the pump speed reaches 100%, the system is rejecting all the heat possible. However, it is not likely that the temperature setpoints in these tests would ever be used in actual operations, as they are not recommended by the manufacturer.

On the remaining Tests #5-#8, the CarnotJetTM system easily maintained stable rack coolant temperatures at setpoint. These tests used setpoints that are most likely to be encountered in the typical installation.

TABLE 6 TEST RESULTS DATA SUMMARY*

* Average values

Test No.

Server

Power

(kW)

Oil

Temp

(°C)

Water

Temp

(°C)

Coolant

Oil Pump

Power

(kW)

CTE

Flow

(GPM)

CTE

Pump

Power

(kW)

LMTD

(°C)

Oil Pump

(kW/ton)

CTE

(kW/ton)

Total

GRC

Power

(kW)

Total

GRC

(kW/ton)

1 69.81 45.1 29.0 0.61 121.6 2.04 10.34 0.031 0.103 2.65 0.134

2 69.80 46.8 32.0 0.80 128.2 2.39 9.45 0.040 0.120 3.19 0.161

3 69.77 35.1 18.0 0.86 128.5 2.40 10.93 0.043 0.121 3.26 0.164

4 69.78 40.1 24.0 0.83 127.8 2.37 10.40 0.042 0.119 3.20 0.161

5 69.82 45.1 24.0 0.29 91.3 0.88 13.48 0.015 0.044 1.17 0.059

6 69.84 45.1 18.0 0.15 72.5 0.45 17.53 0.007 0.022 0.59 0.030

7 69.84 40.2 18.0 0.33 93.5 0.96 14.34 0.017 0.048 1.29 0.065

8 44.83 40.0 18.0 0.08 54.4 0.19 14.35 0.006 0.015 0.27 0.021

23


8.1.2 ESTIMATED ENERGY AND DEMAND CONSUMPTION

The annual energy and peak demand, and energy cost (assume an annual average cost of $0.14/kWh) of the GRC CarnotJet oil pump with a CTE water pump is estimated to be:

TABLE 7 SUMMARY OF ANNUAL ENERGY AND DEMAND CONSUMPTION

8.1.3 ESTIMATED ENERGY AND DEMAND SAVINGS

The annual energy and peak demand, and energy cost savings (assume an annual average cost of $0.14/kWh) of the GRC CarnotJet oil pump with a CTE water pump, compared to the baseline for a Large Data Center with Level II air-management scheme, hot and cold isle containment, ducted-return, with a constant speed CRAH fan, is estimated to be:

TABLE 8 SUMMARY OF ENERGY AND DEMAND SAVINGS

The demand savings are 100% coincident with the utility peak demand period, as the pump and fan equipment operates at 100% duty cycle. This is due to the data center operating at a constant cooling load throughout the entire year, 24/7/365, and the equipment associated with the energy savings operating independent of outdoor ambient condition. The pump and fan equipment are located inside the data center, a data center’s cooling load is driven the constant IT load, not ambient conditions per the baseline document. The demand savings come from pump power instead of fan power.

8.1.4 APPLICABILITY TO RETROFIT AND NC PROGRAM

The technology currently fits into both the Customized Retrofit and the Customized New Construction Programs. There are two main schematic designs for the technology’s implementation, depending upon the facility’s piping into chilled water or condenser water. In this technology assessment it was into the chilled water, and therefore the energy savings came from pump power replacing fan power. Key variables to measure and/or record include: server (IT) power, oil pump power, and CRAH fan power.

In a situation where the technology is plumbed into the condenser water, additional variables should be recorded. Determining which are most crucial is beyond the scope of this assessment, see Section 9.7 “Areas for Further Study.” The following list of variables is that which should be included, but not limited to, for further investigation: chilled water temperature (supply and return), condenser water temperature (supply and return), chiller power, chilled water pump power, condenser water pump power, cooling tower fan power, and outside air temperature (dry and wet bulb).

Demand 2.00 (kW)

Total Energy 17,520.00 (kWh/yr)

Total Cost 2,452.80$ ($/yr)




24


INCREMENTAL COST 8.2The incremental cost of the technology is estimated to be $55,154. The baseline project cost is from the 2013 RSMeans Mechanical Cost Data, adjusted for Oakland, California location, and the assessment technology cost is from the actual project cost.

TABLE 9 BASELINE COSTS

TABLE 10 TECHNOLOGY ASSESSMENT COSTS

8.2.1 TYPICAL PROJECT COSTS

The project cost for this technology assessment is very site specific, based upon the wiring, piping, equipment, labor, and site peculiarities. To address this, the vendor has provided the following example pricing.

TABLE 11 TYPICAL PROJECT COSTS

Baseline Costs Mat'l Labor Tax SubTotal

1 20 ton Computer Room Unit 45,353$ 4,221$ 4,309$ 54,000$

for water cooled system, not incl cond, water supply, CT

Contingency, Overhead/Profit, Design, Misc 35% 19,000$

Shipping (estimated) 5% 2,700$

Total 75,700$

Technology Assessment Project Costs SubTotal

Green Revolution Cooling 4x42U CarnotJet (~16 kW/rack) and installation $121,876

tanks, oil coolant, overhead hoist, tank covers, graphics, tax and shipping

piping (and installation) between the main chilled water line and the GRC $8,978

Total $130,854

25


EXPECTED USEFUL LIFE 8.3The expected useful life of the technology as a whole is estimated to be 15 years. The major equipment included in the system is shown below, with the DEER 2011 Expected Useful Life (EUL).

TABLE 12 EXPECTED USEFUL LIFE

DATA ANALYSIS 8.4

8.4.1 GRC CARNOTJETTM SYSTEM COOLING EFFECTIVENESS

During six six tests, Tests #1, and #4 through #8, which used setpoints that are recommended by the manufactuere, and most likely to be encountered in a typical installation, the CarnotJetTM was able to maintain stable rack coolant temperatures at setpoint.

Test #1 combined the full server load of approximately 69.8 kW with rack coolant temperature setpoint

of 45°C and cooling water temperature setpoint of 29°C. This cooling water temperature is expected to

be capable of being maintained by a cooling tower without the use of a chiller in many areas in CA.

Under these conditions the system’s pumps ran at nearly full speed but the worst case error in rack the

coolant temperature, the biggest difference between setpoint and recorded values, was less than 1°C

TABLE 13 TEST #1 DATA SUMMARY

Test #2 was designated as a worst case condition. During this test cooling water was set to 32°C and the

heat load and rack coolant temperature were kept the same as those in Test #1. A cooling water

temperature of 32°C is considered extremely high and was tested to verify what may happen in the

event of a cooling water system failure (assuming a cooling tower failure). Under these conditions the

system’s pumps ran at full speed continuously and rack coolant temperature could not be maintained.

Equipment EUL Assumptions

water loop pump 15 none - directly from DEER 2013

oil loop pump 15 same as water loop pump

coolant infinite from vendor

racks infinite from vendor

controller 15 same as other mechical controllers in DEER 2013

Test 1

Coolant

Oil Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 29.00 45.00

Min 0.611 1.834 69.51 0.877% 2.632% 117.23 28.51 45.05 10.0 91.8%

Max 0.819 2.408 70.24 1.173% 3.452% 128.57 29.45 45.23 10.7 97.6%

Avg 0.707 2.042 69.81 1.013% 2.925% 121.56 29.00 45.13 10.3 94.7%

SdtDev 0.030 0.086 0.118 4.38E-04 1.23E-03 1.71 0.254 0.044 0.2 0.07

Rel StdDev 4.31% 4.19% 0.17% 4.32% 4.21% 1.41% 0.88% 0.10% 0.0 1.6%

26


TABLE 14 TEST #2 DATA SUMMARY*

* Average Pump Speed was 100% during Test #2 indicating that the system was at or beyond the system capacity to maintain rack coolant temperature.

Test #3 was designed to test system response at low temperatures. Both the cooling water temperature

setpoint and the rack coolant temperature setpoint were minimized in this test. Heat load was

maintained at 69.8 kW. As a result of these low temperatures the cooling pumps ran at full speed and

rack coolant temperature did not get down to within 4°C of the setpoint. This test showed the

limitations of the system to produce low coolant temperatures when the heat load (servers) is high.

TABLE 15 TEST #3 DATA SUMMARY*

* Average Pump Speed was 100% during Test #3 indicating that the system was at or beyond the system

capacity to maintain rack coolant temperature.

Tests #4 through #7 tested system response under the full system heat load (server load) of 69.8 kW

with a variety of temperature setpoint combinations which may be considered typical practice for this

technology. Test #8 tested what the vendor deemed to be a more likely server load of 44.8 kW at

temperature setpoints that they considered to be a most energy efficient operation. Test results are

shown in Table 8.10 through 8.14.

Test 2

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 32.00 45.00

Min 0.803 2.339 69.61 0.961% 2.765% 119.28 31.35 45.44 7.8 100%

Max 0.826 2.416 69.97 1.186% 3.462% 128.72 32.98 47.24 10.1 100%

Avg 0.813 2.386 69.80 1.165% 3.418% 128.17 32.03 46.79 9.4 100%

SdtDev 0.007 0.020 0.095 1.05E-04 2.70E-04 0.36 0.433 0.484 0.5

Rel StdDev 0.86% 0.83% 0.14% 0.90% 0.79% 0.28% 1.35% 1.03% 0.1

Test 3

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 18.00 30.00

Min 0.829 2.348 69.49 1.186% 3.366% 127.48 17.89 34.80 10.8 100%

Max 0.880 2.448 70.17 1.264% 3.511% 129.29 18.16 35.73 11.5 100%

Avg 0.855 2.403 69.77 1.226% 3.444% 128.47 18.00 34.87 10.9 100%

SdtDev 0.008 0.021 0.114 1.23E-04 3.07E-04 0.39 0.059 0.098 0.1

Rel StdDev 0.97% 0.87% 0.16% 0.98% 0.87% 0.30% 0.33% 0.28% 0.0

27






Test 4

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 24.00 40.00

Min 0.732 2.058 69.48 1.049% 2.949% 121.90 21.62 40.04 9.8 95.0%

Max 0.859 2.468 70.26 1.231% 3.531% 129.65 24.85 40.86 11.8 100.0%

Avg 0.828 2.368 69.78 1.186% 3.394% 127.84 24.00 40.14 10.4 99.4%

SdtDev 0.013 0.052 0.124 1.81E-04 7.56E-04 0.97 0.182 0.057 0.1 0.04

Rel StdDev 1.46% 2.12% 0.18% 1.47% 2.14% 0.75% 0.76% 0.14% 0.0 0.8%

Test 5

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 32.00 45.00

Min 0.164 0.466 69.57 0.235% 0.668% 73.67 23.27 44.91 12.8 65.6%

Max 0.401 1.172 70.17 0.574% 1.679% 100.72 24.97 45.60 14.0 77.0%

Avg 0.294 0.880 69.82 0.421% 1.260% 91.32 24.00 45.12 13.5 70.1%

SdtDev 0.024 0.072 0.113 3.51E-07 1.04E-06 2.61 0.280 0.070 0.2 0.10

Rel StdDev 6.09% 6.17% 0.16% 0.01% 0.01% 2.59% 1.17% 0.16% 0.0 2.9%

Test 6

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 32.00 45.00

Min 0.124 0.379 69.53 0.177% 0.543% 68.68 17.08 44.87 17.2 45.6%

Max 0.396 1.249 70.34 0.565% 1.782% 102.91 18.30 45.40 18.3 65.4%

Avg 0.147 0.447 69.84 0.211% 0.639% 72.52 18.00 45.11 17.5 54.5%

SdtDev 0.019 0.058 0.119 2.74E-04 8.21E-04 2.49 0.123 0.087 0.1 0.22

Rel StdDev 4.84% 4.61% 0.17% 4.84% 4.61% 2.42% 0.68% 0.19% 0.0 8.0%

Test 7

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 18.00 40.00

Min 0.219 0.579 69.55 0.313% 0.827% 79.30 16.95 38.71 13.4 62.8%

Max 0.855 2.484 70.33 1.223% 3.552% 129.93 19.65 44.52 16.3 77.0%

Avg 0.330 0.958 69.84 0.472% 1.372% 93.47 18.02 40.20 14.3 70.5%

SdtDev 0.100 0.294 0.116 1.44E-03 4.21E-03 7.21 0.218 0.380 0.3 0.07

Rel StdDev 11.74% 11.83% 0.17% 11.75% 11.84% 5.55% 1.21% 0.95% 0.0 0.0%

28



As expected the GRC CarnotJetTM system pump operations appear to be correlated with server load and the temperature difference available at the heat exchanger. Of particular interest is the combination at which the pump speed reaches 100%. The CarnotJetTM system relies on varying pump speed to maintain rack coolant temperature. When the pump is at full speed the CarnotJetTM system is no longer considered to be controlling rack coolant temperature, and is beyond its capacity.

A predicted speed may be calculated using multivariate linear regression based on: 1) server load and 2) logarithmic mean temperature difference at the heat exchanger, with a relatively high degree of accuracy. Table 21 Test Data and Regression Analysis Predicted Pump Speed shows the predicted value of pump speed from this regression analysis. Details of this analysis can be found in the Appendix.

TABLE 21 TEST DATA AND REGRESSION ANALYSIS PREDICTED PUMP SPEED

Test Server

Load (kW) LMTD

(°C) Recorded

Pump Speed

Predicted Pump Speed

y = ƒ (kW, LMTD)

1 69.81 10.3 0.95 0.97

2 69.80 9.5 1.00 1.02

3 69.77 10.9 1.00 0.93

4 69.78 10.4 0.99 0.96

5 69.82 13.5 0.70 0.77

6 69.84 17.5 0.54 0.52

7 69.84 14.3 0.71 0.72

8 44.83 14.4 0.42 0.42

8.4.2 ENERGY CONSUMPTION AND SAVINGS

As described in the project scope, no pre-existing condition data logging was performed. Therefore the energy savings is compared to a baseline for a Large Data Center, Level II air-management scheme with hot and cold isle containment and ducted-return with constant speed fans, from the PG&E Energy Efficiency Baselines for Data Centers document.

Test 8

Cooling Oil

Pump

Power

(kW)

CTE Pump

Power

(kW)

Server

Power (kW)

Oil

Pump/Server

Power

CTE

Pump/Server

Power

CTE Flow

calculated

Water

Temp

(°C)

Oil

Temp

(°C)

LMTD

(°C)

Pump

Speed

Setpoint - - - - - - 18.00 40.00

Min 0.066 0.083 44.56 0.147% 0.184% 41.04 14.76 39.82 12.4 40.0%

Max 0.093 0.254 45.18 0.207% 0.565% 59.97 20.72 40.36 15.6 45.0%

Avg 0.077 0.194 44.83 0.171% 0.432% 54.42 18.00 40.10 14.3 41.8%

SdtDev 0.004 0.039 0.116 9.02E-05 8.66E-04 3.99 0.953 0.101 0.6 0.06

Rel StdDev 4.35% 15.28% 0.26% 4.37% 15.32% 6.65% 5.29% 0.25% 0.0 2.8%

29


TABLE 22 BASELINE AIR MANAGEMENT

The baseline energy consumption will be taken with a load factor of 1.0.

Baseline annual energy consumption

For an annual average IT load of 69.8 kW the CRAH airflow is:

Airflow = (kW of IT load *3,412 Btu/hr-kW ) / (1.08 * degrees F of operating airside Delta-T)

Airflow = (69.8*3,412) / (1.08*13)

Airflow = 16.963 cfm

The CRAH fan power is calculated as:

CRAH Coincident Fan Power = cfm * fan airflow efficiency metric * duty cycle

CRAH Coincident Fan Power =16.963 * 1,508 * 100%

CRAH Coincident Fan Power = 11.25 kW * 100%

For a data center operating 8,760 hours per year, the annual energy consumption is estimated to be:

Baseline Annual Energy Consumption = 11.25 kW *8,760 hr/yr

Baseline Annual Energy Consumption = 98,550 kWh/yr

Technology Assessment annual energy consumption

For an annual average IT load of 69.8 kW, from the logged data the average GRC CarnotJet and CTE power is:

GRC Coincident Power = (oil pump power + CTE pump power) * duty cycle

GRC Coincident Power = (0.52 kW + 1.48 kW) * 100%

GRC Coincident Power = 2.00 kW

For a data center operating 8,760 hours per year, the annual energy consumption is estimated to be:

GRC Annual Energy Consumption = 2.00 kW *8,760 hr/yr

GRC Annual Energy Consumption = 17,520.0 kWh/yr

Operating

Airside

Delta-T

Fan

Airflow

Efficiency

Metric

(F) (cfm/kW)

1,508

Air Management Type

Hot Aisle/Cold

Aisle, Ducted

Return

II13

30


The GRC Peak Power consumption was taken from the logged data during the “Extreme Summer” test name, and is

GRC Coincident Peak Power = (peak oil pump power + peak CTE pump power) * duty cycle

GRC Coincident Peak Power = (0.52 kW + 1.48 kW) * 100%

GRC Coincident Power = 2.00 kW

Estimated Annual Energy and Peak Demand Savings

Energy savings come from the GRC CarnotJet oil pump and CTE pump replacing the CRAH fan power. The Estimated Annual Energy Savings between the GRC and baseline systems is:

Estimated Annual Energy Savings = Baseline – GRC

Estimated Annual Energy Savings = 98,550 kWh/yr - 17,520 kWh/yr

Estimated Annual Energy Savings =81.030 kWh/yr

The Estimated Peak Demand Savings between the GRC and baseline systems is:

Estimated Coincident Peak Demand Savings = Baseline – GRC

Estimated Coincident Peak Demand Savings = 11.25 kW – 2.00 kW

Estimated Coincident Peak Demand Savings = 9.25 kW

31


EVALUATIONS 9

COOLING PERFORMANCE 9.1The GRC CarnotJetTM system was shown to be capable of maintaining the rack coolant temperature at setpoint in all but the most extreme test conditions. However, during two of the tests, Test #1 and Test #4, the CarnotJetTM system was shown to approach its capacity to maintain coolant temperature. During these two tests the average recorded pump speed was 95% and 99%, respectively. Because the CarnotJetTM system relies on varying pump speed to maintain rack coolant temperature, if the pump speed reaches 100% the system can no longer be considered to be controlling rack coolant temperature, and is beyond its capacity.

TABLE 23 TEST DATA SUMMARY

Test Name

Rack Coolant Temp

Setpoint (°C)

Cooling Water Temp

Setpoint (°C)

Avg. Recorded

Rack Coolant Temp (°C)

Avg. Recorded Cooling

Water Temp (°C)

Average Total

Server Load (kW)

LMTD (°C) Recorded

Pump Speed (%)

1 45 29* 45.1 29.0 69.81 10.3 95%

2 45 32 46.8** 32.0 69.80 9.5 100%

3 30 18 35.1** 18.0 69.77 10.9 100%

4 40 24 40.1 24.0 69.78 10.4 99%

5 45 24 45.1 24.0 69.82 13.5 70%

6 45 18 45.1 18.0 69.84 17.5 54%

7 40 18 40.2 18.0 69.84 14.3 71%

8 40 18 40.0 18.0 44.83 14.4 42%

* Product’s claimed maximum incoming cooling water temperature setpoint. **Rack coolant temperature was not maintained at setpoint.

ENERGY PERFORMANCE 9.2The energy consuming equipment contained in the installed GRC CarnotJetTM system consists of two separate pumps, a GRC CarnotJetTM system Pump Module coolant pump (oil pump) and a GRC CarnotJetTM Cooling Tower Emulator pump (water pump). Over the course of the testing the electrical demand of these two pumps was measured and the average value was calculated to be 2.00 kW. As a percentage of the average server (IT) power the total average GRC CarnotJetTM electrical power demand was found to be 2.94%, or about 0.104 kW of pump power per ton of heat rejected. The annual energy and peak demand, and energy cost (assume an annual average cost of $0.14/kWh) is estimated to be:

TABLE 24 SUMMARY OF ANNUAL ENERGY AND DEMAND CONSUMPTION

Demand 2.00 (kW)

Total Energy 17,520.00 (kWh/yr)

Total Cost 2,452.80$ ($/yr)

32


ENERGY SAVINGS 9.3The annual energy and peak demand, and energy cost savings (assume an annual average cost of $0.14/kWh) of the GRC CarnotJet oil pump with a CTE water pump, compared to the baseline is estimated to be:

TABLE 25 ESTIMATED ENERGY AND COST SAVINGS

The baseline system used for this analysis consists of an air-cooled rack arrangement with hot and cold isle containment, ducted-return, and constant speed CRAH fan.

The demand savings are 100% coincident with the utility peak demand period, as the pump and fan equipment operates at 100% duty cycle. This is due to the data center operating at a constant cooling load throughout the entire year, 24/7/365, and the equipment associated with the energy savings operating independent of outdoor ambient condition. The pump and fan equipment are located inside the data center, a data center’s cooling load is driven the constant IT load, not ambient conditions per the baseline document. The demand savings come from pump power instead of fan power.

FIGURE 12 COINCIDENT PEAK KW DRAW




33


FIGURE 13 ESTIMATED ANNUAL ENERGY CONSUMPTION

OTHER BENEFITS 9.4Not evaluated as part of the scope assigned, but vendor has claimed – increased server efficiency in FLOPs per Watt. LINPACK results are in appendix.

APPLICABILITY TO RETROFIT AND NEW CONSTRUCTION 9.5

PROGRAM The technology currently fits into both the Customized Retrofit and the Customized New Construction Programs. There are two main schematic designs for the technology’s implementation, depending upon the facility’s piping into chilled water or condenser water. In this technology assessment it was into the chilled water, and therefore the energy savings came from pump power replacing fan power. Key variables to measure and/or record include: server (IT) power, oil pump power, and CRAH fan power.

In a situation where the technology is plumbed into the condenser water, additional variables should be recorded. Determining which are most crucial is beyond the scope of this assessment. The following list of variables is that which should be included, but not limited to, for further investigation: chilled water temperature (supply and return), condenser water temperature (supply and return), chiller power, chilled water pump power, condenser water pump power, cooling tower fan power, and outside air temperature (dry and wet bulb).

The technology has significant energy savings potential, a long expected useful life, and a high incremental cost. Based upon conversation with the vendor there appears to be low market penetration to date.

34


MARKET BARRIERS 9.6Based upon conversations with the vendor and facility personnel, the following market barriers were identified:

1) Requires the use of solid state drives, which are expensive

a. typical magnetic style hard drives are not hermetically sealed, and allow mineral oil through their seals which ruins the drive

2) Server Manufacturers do not yet offer “off the shelf” servers to submerge into mineral oil. Currently each server requires (at minimum):

a. Removal of thermal paste (clouds oil & could plug orifices)

b. Removal of air-cooling fan

3) Obtaining Server Manufacturer support to fully honor the server warranty for those modified and submerged. Currently the vendor is working with numerous server manufacturers’ to address this issue, and the server manufactuers are aware of the potential interest by customers.

4) Network gear manufacturers (switches, network interface cards, etc) lagging further behind server manufacturer.

5) Meeting criteria for return on investment.

6) Site plumbing specifics

a. Availability of condenser water within the data center is preferred

b. If this is not available, chilled water will suffice, but will not result in as significant energy savings. This was the case in the technology assessment, and no chiller or chilled water pump savings was attained.

It was asked if any barrier due to maintenance or the CarnotJet or servers by facility personnel was a barrier. Real Estate Facility personnel stated that they were capable of maintaining the technology assessed. IT personnel stated that the servers require extra time to remove and maintain, as the oil is dripped from the server, and it needs to be packaged in a plastic bag if it is transported but that also was not a barrier.

AREAS FOR FUTURE STUDY 9.71) Detailed investigation of the potential energy and coincident demand savings with the cooling water

for the GRC system coming from condenser water instead of a chiller. This would result in fan, pump, and chiller savings.

2) Market penetration and portfolio energy savings for product design and workpaper.

3) Evaluate any energy savings claim for increased server efficiency in FLOPS per Watt. Vendor claims reduced server power and increased computation due to a lower, and more constant, chip and processor temperature.

35


4) New Construction application – There is significant savings in Data Center Infrastructure Management through this type of installation. The avoided costs of chiller, pumps, piping, wiring, controls, building, and workers to house such reductions could be significant.

5) Similar study with all IT load from servers, instead of partially from load banks.

FINANCIAL ANALYSIS 9.8The financial analysis is based upon the estimated energy savings and incremental cost shown in the Results Section. It assumes an annual average electrical energy cost of $0.14/kWh. The technology is eligible under the Customized Retrofit Program for Incentive at a rate of $0.08/kWh and $100/peak-kW.

TABLE 26 FINANCIAL ANALYSIS

The financial analysis does not take into account any costs or savings due to maintenance, infrastructure (central plant), or server modification.


Incremental Cost $55,154 ($)

Simple Payback 4.86 (years)

kW INCENTIVE $925 ($)

kWh INCENTIVE $6,482 ($)

Total INCENTIVE $7,407 ($)

Simple Payback with INCENTIVE 4.21 (years)

36


RECOMMENDATIONS 10

ENGAGEMENT IN ENERGY EFFICIENCY PROGRAMS 10.1The assessment showed that the GRC CarnotJetTM system provides for energy and peak coincident demand savings over the incumbent technology and we recommend its adoption into EE programs. The technology currently fits into the Customized Retrofit and Customized New Construction Programs. For an installation similar to the one installed in this project, which included a GRC Cooling Tower Emulator system, the key variables in determining energy savings are server (IT) power, GRC CarnotJetTM oil pump power, GRC CarnotJetTM CTE pump power, and the facility’s pre-existing CRAH fan power. The technology has significant energy savings potential, a long expected useful life, and a high incremental cost. Based upon conversation with the vendor there appears to be low market penetration to date. The current DEER compliance tool does not address this technology. There are still a number of suggestions that the Emerging Technologies Team suggests for further investigation, shown in the next section.

SUGGESTION FOR FURTHER INVESTIGATION 10.21) Determine energy and coincident demand savings potential, as well as incremental cost for re-piping

a facility so that the cooling water comes from condenser water, instead of a chiller, for a brown-field retrofit of the numerous air flow scenarios. A rough estimate of the energy consumption for a four rack system, based upon load factor of 1.0 and equipment energy consumption metrics from the PG&E Data Center Baseline document is:

37


FIGURE 14 THEORETICAL ENERGY CONSUMPTION FOR REPIPING CONDENSER WATER TO GRC CARNOTJETTM

FIGURE 15 THEORETICAL ENERGY SAVINGS FOR REPIPING CONDENSER WATER TO GRC CARNOJETTM

2) Market penetration and portfolio energy savings for product design, development of workpaper for a prescriptive, or deemed, rebate instead of a customized incentive. This would be a first step in having the compliance tool of DEER address this technology.

3) Evaluate any energy savings claim for increased server efficiency in FLOPS per Watt. Vendor claims reduced server power and increased computation due to a lower, and more constant, chip and processor temperature.

4) Data Center Infrastructure Management (DCIM) Investigation. In a new construction, green-field type installation of a CarnotJet system, the temperature of the oil exiting the CarnotJet system is high enough to allow for the server heat to be rejected in a cooling tower alone. No vapor-compression refrigeration system is necessary, greatly reducing the number of heat transfer mediums, mechanical equipment, and its associated costs. There would be only two heat transfer mediums:

a. Ambient air to condenser water

b. Condenser water to oil, in which the server is submerged A significant amount of infrastructure first and operational cost savings, as well as significant energy savings may result from this. The only energy consuming equipment would be a cooling tower fan, condenser water pump, and the CarnotJet oil pump.

5) Similar study with all IT load from servers, instead of partially from load banks.

38


APPENDICES 11During testing servers running specially designed test scripts were analyzed for CPU performance. The data was made available to the team..

TABLE 27 LINPACK DATA SUMMARY

GFLOPS WATTS GFLOPS/W

Test Min 115.7203 148 0.7804

Test Max 176.7660 176 1.0066

Test AV 175.8174 177 0.9927

Summer Min 173.2292 178 0.9708

Summer Max 176.3910 179 0.9867

Summer AV 175.9358 178 0.9860

Extreme Min 115.7203 148 0.7804

Extreme Max 176.1614 179 0.9855

Extreme AV 127.7268 157 0.8143

Winter Min 174.9711 177 0.9882

Winter Max 176.7660 176 1.0066

Winter AV 176.4688 176 1.0002

Typical Min 174.7642 178 0.9819

Typical Max 176.5352 177 0.9987

Typical AV 176.1553 177 0.9931

Add 1 Min 175.9446 178 0.9873

Add 1 Max 176.3685 178 0.9911

Add 1 AV 176.1503 178 0.9899

Add 2 Min 175.0657 177 0.9865

Add 2 Max 176.4659 178 0.9911

Add 2 AV 176.1635 178 0.9902

Add 3 Min 175.8122 177 0.9961

Add 3 Max 176.5726 177 0.9983

Add 3 AV 176.2568 177 0.9977

Winter2 Min 176.0688 176 1.0015

Winter2 Max 176.7164 175 1.0084

Winter2 AV 176.4477 176 1.0025

Typical Load Min 173.8807 178 0.9757

Typical Load Max 176.4915 178 0.9936

Typical Load AV 176.1230 178 0.9911

Test 3

Test 2

Test 1

Overall

Test 8

Test3

Test 7

Test 6

Test 5

Test 4

39


TABLE 28 LOGGED DATA FROM TESTS (TRUNCATED)

Logger DLP06 PEC 123 PEC206 Powersight Total Load CW Flow

Total GRC

power

Channel Channel 1 Channel 2 Channel 3 Channel 4 Channel 1 Channel 5 Channel 4 Channel 6 Channel 4 Channel 5 Channel 5 Channel 1 Channel 5

Circuit K3 7,8 K3 9,10 K3 11,12 K3 13,14 L1 6,7 L1 8,9,10 1L (3) 4,5,6 1L3 9,10 K2 11,12 K (3) 1,2,3 Total K4 Panel L3 7,8 K (3) 4,5,6 L(3) 1,2,3 Excl. Pumps

Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW Avg. KW gpm

1ph2wire 1ph2wire 1ph2wire 1ph2wire 1ph2wire 3ph Power 3ph Power 1ph2wire 1ph2wire 3ph Power 3ph Power 1ph2wire 3ph Power 3ph Power

Load notes 2nd Oil Pump Pri Oil Pump Rack 1 GRC H2O Pri Rank 2 Rack 1 Rack 1 & 2

Time Stamp

10/4/2013 15:00 3.446 3.434 3.521 3.398 0 0.713 1.709 4.789 2.029 8.695 27.917 3.42 3.975 5.3816 69685.6 2.742

10/4/2013 15:05 3.446 3.434 3.519 3.395 0 0.638 1.71 4.797 1.89 8.664 27.917 3.42 3.979 5.388 69669 2.528

10/4/2013 15:10 3.447 3.436 3.52 3.397 0 0.67 1.709 4.68 1.951 8.73 27.913 3.42 3.972 5.3584 69582.4 2.621

10/19/2013 14:15 0.00 0.00 1.85 0.00 0.00 0.08 4.94 0.00 0.24 8.95 19.58 0.00 4.03 5.51 44862 0.325

10/19/2013 14:20 0.00 0.00 1.85 0.00 0.00 0.08 4.90 0.00 0.25 8.84 19.58 0.00 4.02 5.52 44710 0.329

10/19/2013 14:25 0.00 0.00 1.85 0.00 0.00 0.07 5.04 0.00 0.13 8.80 19.58 0.00 4.02 5.52 44811 0.197

10/19/2013 14:30 0.00 0.00 1.85 0.00 0.00 0.07 4.78 0.00 0.17 8.92 19.58 0.00 4.02 5.52 44674 0.245

10/19/2013 14:35 0.00 0.00 1.85 0.00 0.00 0.08 4.84 0.00 0.20 8.88 19.58 0.00 4.02 5.52 44695 0.284

10/19/2013 14:40 0.00 0.00 1.85 0.00 0.00 0.09 4.89 0.00 0.25 8.98 19.58 0.00 4.02 5.52 44838 0.33

10/19/2013 14:45 0.00 0.00 1.85 0.00 0.00 0.07 4.97 0.00 0.17 8.93 19.58 0.00 4.02 5.51 44877 0.24

10/19/2013 14:50 0.00 0.00 1.85 0.00 0.00 0.07 4.85 0.00 0.17 8.94 19.59 0.00 4.02 5.53 44778 0.235

10/19/2013 14:55 0.00 0.00 1.85 0.00 0.00 0.08 4.91 0.00 0.23 8.94 19.58 0.00 4.02 5.51 44803 0.302

10/19/2013 15:00 0.00 0.00 1.85 0.00 0.00 0.08 4.79 0.00 0.14 8.82 19.58 0.00 4.03 5.50 44572 0.222

DLP05

GR

C T

YP

ICA

LSU

MM

ERTe

st N

ame DPL01 DPL02 DPL04

40


FIGURE 16 GRC ONLINE SYSTEMS MONITOR DATA

41


FIGURE 17 SENTILLA ONLINE DASHBOARD

42


REFERENCES 12Pacific Gas & Electric Company. (2013). Energy Efficiency Baselines for Data Centers, 2nd Ed. Oakland,

CA:: Integral Group. Retrieved 01 27, 2014, from http://www.pge.com/includes/docs/pdfs/mybusiness/energysavingsrebates/incentivesbyindustry/hightech/data_center_baseline.pdf

Tulkoff, C. a. (2013). Improved Efficiency & Reliability for Data Center Servers Using Immersion Oil Cooling. Electronic Systems Technologies Conference & Exhibition. Las Vegas, NV.

United States Deparment of Energy. (2006). Quick Start Guide to Increase Data Center Energy Efficiency 1st Ed. Washington DC: General Services Administration. Retrieved 01 27, 2014, from http://hightech.lbl.gov/documents/data_centers/Quick-Start-Guide.pdf

U.S. Environmental Protection Agency ENERGY STAR Program (August 2, 2007), “Report to Congress on Server and Data Center Energy Efficiency Public Law 109-431.” 1st Ed, http://hightech.lbl.gov/documents/data_centers/epa-datacenters.pdf (accessed 1/27/14): U.S. Environmental Protection Agency.

http://hightech.lbl.gov/documents/data_centers/epa-datacenters.pdf

Submersion Cooling Evaluation - ETCC · project, contact [email protected] or [email protected]. Legal Notice...

Documents

Transcript of Submersion Cooling Evaluation - ETCC · project, contact [email protected] or [email protected]. Legal Notice...