The Role of Computational Science in Energy Efficiency and Renewable

Energy Research

Steve Hammond, DirectorComputational Sciences Center

National Renewable Energy Laboratory

PACT02September 2002

Operated for the U.S. Department of Energy by Midwest Research Institute • Battelle • Bechtel

Outline

• Why NREL?• Energy Scene• NREL mission and programs• Computational Sciences at NREL• Architecture and Programming Model

Observations

Why NREL?

The Energy Scene

• Viewed over decades, changes to energy markets are sweeping and fundamental including impacts to economy, environment, and national security.

• Just after WW II, homes were heated with coal, traveled on steamships, and leaded fuel.

• Today, heat with natural gas, travel on jet planes, use unleaded fuel.

• In 1950, America imported 5% of its oil.• Today, U.S. imports 62% of its “fossil fuel.”

Dependence on Foreign OilU.S. Transportation use of Petroleum

1970 1980 1990 2000 2010 2020

Mill

i on s

of B

a rr e

ls p

er D

a y

TodayDomestic Petroleum Usage

Imports

Actual: Annual Energy Review 2000 Tbls 1.2, 5.1 and 5.12Forecast: Annual Energy Outlook 2002 Tbls 7 and 11Split between Autos and Lt Truck: Transportation Energy Data Book Edition 21 Tbl 2.6Updated April 2002

0

5

10

15

20

25

30

Affluence Requires Energy – Poverty Breeds Global Insecurity

0.1

1

10

100

0.1 1 10 100 1000

Energy Consumption Per Capita ('000 BTU/person)

GD

P P

er C

apit

a ($

000/

per

son

)

Poverty

Affluence

Burkina Faso

United Kingdom

Mexico

Bangladesh

China

Poland

South Korea

United StatesFranceJapan

Source: Energy Information Administration, International Energy Annual 2000 Tables E1, B1, B2; Gross Domestic Product per capita is for 2000 in 1995 dollars. Updated May 2002

El Salvador Russia

Renewable Energy “factoids”

• North Dakota has enough wind to supply 35% of the U.S. electrical demand.

• The sunlight falling on the U.S. in one day contains more than twice the energy we consume in an entire year.

• Renewable energy is plentiful.• Continued R&D is necessary to ensure that

“renewables” are efficient, reliable, and affordable.

National Renewable Energy Laboratory DoE Lab, located in Golden, CO Only national laboratory dedicated to renewable

energy and energy efficiency R&D Research spans spectrum of fundamental science

to technology solutions and assessment Collaboration with industry and university partners

is a hallmark About 1000 employees, and approx. $200M/year

NREL Mission StatementNREL develops renewable energy and energy efficiency technologies and practices, advances related science and engineering, and transfers knowledge and innovations to address the nation's energy and environmental goals.

Major NREL Technology Thrusts

Supply Side Wind Solar

- Photovoltaics- Concentrating Solar Power

- Solar Buildings

Biomass- Biopower

- Biofuels - Biorefinery

Geothermal Hydrogen Superconductivity Distributed Energy

Demand SideFreedomCAR and VehicleTechnologies - Hybrid Vehicles - Fuels Utilization

Building Technologies- Zero Energy Buildings

Federal Energy ManagementAdvanced Industrial TechnologiesCrosscutting

Basic Energy Science

Analytical Studies

Computational Science

Outdoor Test Facility

National Center for Photovoltaics Facilities

Outdoor Test Area

Solar Energy Research Facility

Solar Photovoltaics• In the last 25 years, cost of

electricity from PV has dropped from $1/kWh to about 20¢/kWh today.

• Strong efforts, government support in Japan and Germany

• U.S. market share 40%• R&D: fundamental science of

materials, advanced solar cells and processes, scale-up, lower cost

Sacramento Municipal Utility District (01026)

Source: Technology Opportunities to Reduce U.S. Greenhouse Gas Emissions, October 1997

National Wind Technology Center • Facility for NREL scientists and

the U.S. wind energy industry to conduct research, engineering and development of advanced wind technology

• Unique facilities– 280 acres of ideal wind

research land– 16 outdoor turbine test pads– Structural test facilities for

turbine blades and rotors– Test facilities for wind turbine

components

Wind turbine structural test facilities (04364)

Wind Energy

• In the last 25 years, cost of wind energy has decreased from 40¢/kWh to about 4¢/kWh.

• 2010 goal is 2–3¢/kWh• Strong European research

efforts.• R&D: improvements in

turbine designs, structural dynamics, lower cost

Green Mountain Power Wind Plant, Vermont (05592)

Source: Technology Opportunities to Reduce U.S.Greenhouse Gas Emissions, October 1997

GE WindEnergy 3.6 MW Prototype Turbine in Spain

GE WindEnergy 3.6 MW Prototype Turbine in Spain

Boeing 747-200

Buildings Research Activities

Analyzing air leaks in HVAC equipment using infrared

cameras

Investigation of boundary layer effects in transpired

collectors

Analyzing energy performance

Solar Buildings Program

• Combine solar energy technology with energy-efficient construction techniques.

• Help create a new generation of cost-effective buildings that have zero net annual need for non-renewable energy.

NREL Bioenergy Facilities

Field Test Laboratory Building Bioethanol Process Development Unit

ThermochemicalProcess DevelopmentUnit

Alternative Fuels User Facility

Wood chipsSwitch grass

Poplars

Municipal solid wasteSugar cane residue

Alfalfa

Biomass Sources

Biomass Electric

• Direct combustion – 7500 MWe installed capacity

• Cofiring (wastes) – demonstrations

• Biomass gasification combined cycle (energy crops) – in development

• Regrowing biomass (energy crops) results in very low or zero net CO2 emissions

• R&D: ash chemistry and deposition, advanced gas turbine technologies

Source: Technology Opportunities to Reduce U.S. Greenhouse Gas Emissions, October 1997

Biomass Transportation Fuels• Ethanol costs have plummeted:

– 1980’s: $4/gal– Current: $1.22/gal estimate– 2010: $0.67/gal estimate

• Develop energy crops for bulk fuel

• Employ biochemical and thermochemical processing

• Displacing gasoline with ethanol in light-duty vehicles gives 90% reduction in carbon emissions

• R&D: low-cost production of enzymes, development of microorganisms,energy crop advances

Source: Technology Opportunities to Reduce U.S. Greenhouse Gas Emissions, Oct 1997

Distributed Energy Resources

Goal• Increase sources of energy production –

reduce dependence on large critical “targets.”

• Triple distributed power generation by 2010R&D Needs• Interconnection standards development and

controlled interconnection testing critical to market development

• Distribution systems research– Distribution system behavior (modeling,

testing)– Distributed command and control systems

development

Transportation Technologies & Systems

• R&D in advanced vehicle and fuel technologies.

• Many small efficiencies add up quickly– Smart tires– Reduced auxiliary loads

• Strong partnerships with public and private organizations.

• Goal: reduce dependence on imported oil and improve air quality.

5915

19M illions of

G allons U sedfor AC per Year

0113227340454567680794907

6

33

64

32

7

20047

844

208

17

137

238

37

117

192

285

75

62

179

19

41

10

225

18

99

157

82146

12

127

77

208

9182

147241

25

69

93

261

907

83472

27798

196

0

National A/C Fuel Use

• 7.1 billion gallons used for air conditioning annually.• Equivalent to 10% of U.S. imported crude oil.

Computational Sciences Challenge

• The application area is compelling and is of increased importance since Sept. 11– Lessened dependence on foreign energy sources

improves homeland security.– Distributed energy reduces risk.– Global affluence and stability requires Energy.

• NREL has a tremendous, dedicated staff.• Very strong theory and experimental focus.• People ask, “Why fix it if isn’t broken?”

Computational Science

Theory is difficult to validate and building prototypes slow and expensive.

Numerical simulation enables the study of complex systems and natural phenomena that would be too expensive or dangerous, or even impossible, to study by direct experimentation.

Computation is an equal and indispensable partner, along with theory and experiment, in the advance of science and engineering.

Vision

• Create a world-class computational science center that enable the best research in energy efficiency and renewable energy research.

• Augment and substantially enhance core laboratory-wide HPC research, capability, and competency in support of NREL’s mission.

Goal for 2005

• Establish 1st class research group– Computational scientists for simulation, data mgmt,

analysis, visualization, and collaborative problem solving environments.

• Integrate CS into NREL mission and national/international renewable energy research community.

• Establish multidisciplinary “Expedition Teams” to solve most pressing energy efficiency and renewable energy problems.

• Establish links with other DoE Labs, C.S. programs.• Establish summer student, post doc, and visitor

program.

Three CS focus areas at NREL

• Simulation: Numerical methods & algorithms– Computational fluid dynamics– Computational chemistry– Structural mechanics

• Data: Currency of the scientific process– Analysis and visualization– Data mining

• Integration: Holistic view– Tools, Systems, and Enterprises– Collaborative Problem Solving Environments

Wind Turbines

• 70m blade challenges:– No wind tunnels

large enough– Shifting winds cause

rotation and blades cut back into own wake

– Structural analysis– Aero-acoustics

• Simulate turbine ensemble

• Incorporate accurate PBL information into simulations

Materials Research

Internationally recognized Solid State Theory Group:

• Study materials and alloys for more efficient photovoltaic systems and solid state lighting.

• Nanostructures – given desired properties, what is the best material?

• Model chemical systems using empirical force fields to describe how molecular potential energies change with atomic coordinates.

• Forces in a molecule can be computed from potential energies. With the atomic forces, the motions of the molecule can be computed by solving Newton's Equations of Motion for all of the atoms in the molecule.

cellulose

H2O

H2O

H2O

H2O

H2O

“Reaction Box”

enzyme

Molecular Mechanical Modeling of Cellulases

CFD - HVAC Systems

• Simulation predicts the distribution of air velocity and temperature using building geometry, vent locations, and fan performance curves.

• Avoid over-engineered system.• Plumes of hot gas and smoke

from fires can be predicted in order to assist in safety analysis

• Additional applications to biohazard or chemical attacks.

Transportation Efficiency

• Passenger comfort is the focus of vehicle climate control. A computational model of the interior of the vehicle yields comfort indicators that allow the design to be improved.

• Study thermal loading.• Optimized climate

control leads to smaller air conditioners and heating systems – greater fuel economy.

partn

er in

fo

(curr

ent,c

oncep

ts)

Packaging/

Visualization

StructuralAnalysis C

om

ponen

t

Data

Optimization

Materials

Lab Partners

Veh

icle

Dat

a

Market

Data

Vehicle

OEMs

Componen

t

Suppliers

New, Better New, Better ComponentComponentss

Cleaner, More Cleaner, More Efficient VehiclesEfficient Vehiclespartner info

(current,concep

ts)

Users

CFD

Veh

icle

D

yn

am

ics

Vehicle

System

Level

Analysis

DigitalDigital

FunctionFunctionalal

VehicleVehicle

energy interactions dataintegrated systems data

Digital Functional Vehicle Wheel

Strategy

• Visit the diverse research groups to assess short-term and long-term needs. Set priorities.

• Cannot go it alone – – Look for collaboration opportunities with university

community.– Develop close ties to other DoE CS programs.

• Seek appropriate funding opportunities.• Initiate student, post doc, & visitor program.• Integrate NREL’s research agenda and CS Initiative

into the national CS infrastructure.

Initial Assessment

• Tremendous opportunity to make a difference.• Most extant modeling and analysis occurs with

greatly simplified models.• Most work currently done on PCs and

workstations• Limited use of machines at other DoE labs.• Solid State Theory, Wind, Molecular

Mechanical Modeling, and Transportation Systems – biggest initial HPC users.

First steps …

• Started March 4th.• IBM SP “starter system” arrived March 28th.• Hired three staff members.• Planning office space, fall occupancy, 9 staff.• Identify key early adopters and set milestones for

early success stories.• Work with other Research Center staff to develop

CS strategic plan and implementation.• Long term - planning new building for FY09

occupancy.

NREL’s Applications

• Complex geometries lead to unstructured grids.

• Heterogeneous computations.• Often grids are adaptively refined (dynamic)

leading to load balancing issues.• Global communications required.

Scientific Computing Observations

• The high performance computing market is a niche market at best.

• With the exception of a few Grand Challenge applications and “publicity stunts,” vast majority of scientific computing now done on 128 PEs or less.– Limited scaling in current application codes.– Better turn-around times - queue wait times

excessive for large PE counts.– Linear charging schemes.

The Case for Shared Memory

• Relatively easy to build shared memory systems with 128 PEs

• What’s so good about shared memory systems?– Much simpler programming model.– Easy to start parallelizing.– Natural load balancing – threads.– Dynamic load balancing nearly impossible with MPI.– Tools, tools, tools, …

Shared Memory Systems

• Need improved hardware and software support for shared memory programming.– Low thread overhead (spawn, synchronization, …)– User control of memory placement (critical in

distributed shared memory, NUMA, systems)

• Application development environment in shared memory systems far superior to distributed memory systems.

• Sun and SGI continued viability???• Multiple processors on a chip could help.

Large Systems

• Suppose that we have efficient shared memory systems.

• Can’t ignore the grand challenge problems.• “Chunky” clusters - build large systems from

clusters of shared memory systems.

• While at NCAR we studied use of large systems for climate modeling.

Message Passing Analysis

• Suppose that you could sustain 16 Gflops on a single shared memory system.

• A sustained Teraflop system could be assembled from a cluster of 128 of these nodes.

• Used 2D decomposition of NCAR’s CCM3.2 (MPI).• For simplicity, we assumed computation and

communication are both perfectly load balanced.

• Counted number of bytes and number of messages per day of simulated climate (2 day - 1 day) per node.

High Resolution Climate Model Message Passing Statistics

NCAR’s CCM3 T170L18 4x32 decomposition, 128 “node” system.

Message Bin Size in Bytes

Total Msgs

Msg Size in Bytes

Total Bytes

0 – 8 25,264 8 2.02E+05

9 – 256 2,208,856 256 5.65E+08

257 – 4096 317,224 4096 1.30E+09

4097 – 65536 1,116,112 65536 7.31E+10

65537 – 524288 1,166,220 524288 6.11E+11

524289 – 1048576

222,976 1048576 2.34E+11

Thousands of Msgs per

Node

39.51 GB per Node

6.70

Interconnect Needs: Thin and Fat Pipes

• Low Latency “Thin Pipes”– Control, Status, – Global Reductions

• High Bandwidth “Fat Pipes”– Bulk data transfer– Data transpositions

• Real world applications need both.• CM5 had the fat tree and the control network.

Summary

• Energy is crucial for prosperity and security.• NREL is America’s premier laboratory for energy

efficiency and renewable energy research.• NREL is establishing a world class Computational

Science capability in support of and commensurate with its other research efforts.

• Improvements in shared memory systems would be beneficial to scientific computing community.

Operated for the U.S. Department of Energy by Midwest Research Institute • Battelle • Bechtel