Dr. Greg HidleyCalifornia Institute for Telecommunications

and Information Technology, UCSD

Project GreenLight: DC Modular Datacenter for Improved Energy Efficiency

http://greenlight.calit2.net

The NSF-Funded GreenLight ProjectGiving Users Greener Compute and Storage Options

• PI is Dr. Tom DeFanti• $2.6M over 3 Years to construct GreenLight Instrument

– Start with Sun Modular Data Center• Sun Has Shown up to 40% Reduction in Energy• Measures Temperature at 5 Levels in 8 Racks• Measures power Utilization in Each of the 8 Racks• Chilled Water Cooling input and output temperatures

– Add additional monitoring at every receptacle– Add web and VR interfaces to access measurement data

• Populate with a variety of computing clusters and architectures– Traditional compute and storage servers– GP GPU arrays and specialized FPGA based coprocessors– DC powered servers

• Turn over to investigators in various disciplines • Measure, Monitor and Collect Energy Usage data

The GreenLight Project: Instrumenting the Energy Cost of Computational Science

• Focus on 5 Communities with At-Scale Computing Needs:– Metagenomics– Ocean Observing– Microscopy – Bioinformatics– Digital Media

• Measure, Monitor, & Web Publish Real-Time Sensor Outputs– Via Service-oriented Architectures– Allow Researchers Anywhere To Study Computing Energy Cost– Enable Scientists To Explore Tactics For Maximizing Work/Watt

• Develop Middleware that Automates Optimal Choice of Compute/RAM Power Strategies for Desired Greenness

UCSD is Installing Zero Carbon EmissionSolar and Fuel Cell DC Electricity Generators

San Diego’s Point Loma Wastewater Treatment Plant Produces Waste Methane

UCSD 2.8 Megawatt Fuel Cell Power Plant Uses Methane

2 Megawatt of Solar Power Cells

Installed

Available Late 2011

Zero Carbon GreenLight Experiment:Direct DC-Powered Modular Data Center

• Concept—Avoid DC to AC to DC Conversion Losses– Computers Use DC Power Internally– Solar and Fuel Cells Produce DC– Both Plug into the AC Power Grid– Can We Use DC Directly (With or Without the AC Grid)?

• DC Generation from Renewable Can Be Intermittent – Depends on Source

• Solar, Wind, Fuel Cell, Hydro– Can Use Sensors to Shut Down or Sleep Computers– Can Use Virtualization to Halt/Shift Jobs

• Experiment Planning Just Starting– Collaboration with Oracle, LBNL, EPRI, Direct Power, Intel, Delta, Emerson – NSF GreenLight Year 2 and Year 3 Funds– CEC and PIER support

DC Fuel Cell 2800kWattsSun Box <200kWatt

Two Rack DC Experiment now Operational

• Proof of concept experiment within context of Project GreenLight• Uses Emerson rectifier as source of DC power => 400VDC• Uses Emerson DC PDUs• Drives 400VDC custom server power supplies (from Delta and

Emerson)• Drives 13 servers

– 6 SunFire X4270 (Nehalem) servers– 6 Intel 2600 (Nehalem) servers– 1 SunFire x4540 storage servers

Transformer Rectifier480VAC

PDU Servers400VDC400VDC

Potential Energy Efficiency Improvements using DC Power– Anticipate reductions in all levels of energy chain from FC to servers using

400VDC • Fuel Cell Inverter (DC to AC)• DC-DC conversion losses (adjust to grid voltage)• 480VAC distribution line losses• 480VAC to 208VAC or 120VAC step down losses• UPS (AC to DC to charge batteries and back to AC) conversion losses• PDU’s (AC voltage stepped down to server requirements)• Inside Power Supplies (rectification and transformation losses)

– Anticipate reduction in cooling loads given heat dissipation of conversions– Anticipate reduction of power utilization by 25% or better (power and cooling)– Anticipate cooling efficiencies of 33% or better

END

Capabilities sneak peak…

GLIMPSEDecision Support System

D E M O2010.08.20

Situational Awareness

“Tap” for details

Dashboard interface

Power utilizatio

n

Multiple perspective

s

Enterprise reach

Datacenter vitals

Input/Output

temperature sampling

Live/Average

data

Live Temperatur

e

Live/average

Fan speeds

Environmentals

Heat Exchangers

Domain specific views

Control elements

Real-time heatmap

Realistic models

Airflow dynamics

Live fan speeds

Airflow dynamics

Heat distribution

Combined heat + fans

Realistic correlation

Heat Trends

Trends over past 24h

Heat exchangers

Hotspot identification

Past changes in airflow

Fan slicesrpm

Potential for failures

Trends over past 24h

Heat distribution

changes

Power spikes

1 minuteresolution

Unused asset

Average load

IT assets

Peak computation

Computation zone

Zoom-in Analysis

2010.08.20

History over several days.

Zoom on desired time range.

Hint on each sample point.

Automatic average area.

Multiple sensors per asset with up to 1 min sampling resolution.

Calit2/UCSD [http://greenlight.calit2.net]

SummaryGLIMPSE – decision support system

» just a sliver of entire GreenLight Enterprise capabilities

» critical to understand the massive amounts of data

» provide real-time situational awareness» monitor & control datacenter assets

GreenLight Enterprise Solution:» federates datacenters» improves performance/watt by selecting most

appropriate architectures» improves energy utilization by scheduling

computation based on heat, energy usage, and load

» reduces cooling needs and improves equipment lifetime