Using Logic Models in Managing Performance of Research and Technology Programs: An example for a...

Using Logic Models in Managing Performance of Research and

Technology Programs:An example for a Federal energy efficiency and

renewable energy program

IAMOT 13th International Conference on Management of Technology

April 4, 2004

Gretchen Jordan, Sandia National LabsJohn Mortensen, Independent Consultant

John Reed, InnovologieGeorge Teather, Independent Consultant

Work presented here was completed for the U.S. DOE Office of Energy Efficiency and Renewable Energy by Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy ( DOE) under Contract DE-AC04-94AL8500. Opinions

expressed are solely those of the authors.

April 4, 2004 2

Presentation Overview

• Managers are under increasing pressure to demonstrate the value of their programs

• Logic models can help programs identify and explain their value

• Logic models can include strategies that represent research, development, demonstration and deployment (RDD&D) stages

• Logic model example for federal energy program• Logic models help programs identify:

– Indicators– Performance targets– Evaluation questions

April 4, 2004 3

There is increasing pressure on RDD&D programs to demonstrate value

U.S. Requirements:• Government Performance and Results Act of

1993 (GPRA) requires agencies to develop:– Strategic plans– Annual performance plans– Annual performance reports

• OMB’s Program Assessment Rating Tool (PART) scores programs on:– Program purpose and design– Strategic planning– Program management– Program results

April 4, 2004 4

Logic models help programs identify value

• A logic model is a diagram or table describing how a program will solve identified problems

• Elements of a logic model include:– Resources (Inputs)– Activities– Outputs– Customers reached– Outcomes

• Short• Intermediate• Long-term

– External influences

April 4, 2004 5

Basic logic model structure

Resources(Inputs)

Activities OutputsShort-TermOutcomes

IntermediateOutcomes(through

customers)

Long-TermOutcomes& ProblemSolution

forCustomersReached

External Influences and Related Programs (mediating factors)

April 4, 2004 6

A logic model example

• A logic model was developed for the U.S. Department of Energy’s Office of Energy Efficiency and Renewable Energy (EERE)

• EERE’s goals are to:– Modernize energy conservation

– Increase energy supplies

– Modernize our critical energy infrastructure

April 4, 2004 7

EERE has 7 strategies for achieving its goals

1. Plan and assess programs

2. Develop and maintain program infrastructure

3. Conduct research

4. Develop technology

5. Demonstrate technology

6. Develop government and market infrastructure

7. Deploy technology

April 4, 2004 8

The 7 strategies cover stages in the RDD&D spectrum and are not necessarily linear

Program planning &

assessment

Conduct research

Developtechnology

Demonstrate technology

Deploy technology

Develop & maintain program

infrastructure

Developing government &

market infrastructure

Feedback Loops

R D D D

April 4, 2004 9

The strategies are represented as “activities” in the draft logic model

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology


infrastructure



Feedback Loops

Inputs

Activities

Outputs

Outcomes

For

April 4, 2004 10

EERE’s draft logic model shows how its strategies/activities are linked to its goals

Inputs

Activities

Outputs

Outcomes

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology

Federal, state & local government fundingPrivate funding, Personnel, Facilities, Past R&D results


infrastructure



Benefit estimates,Priorities identified,

Budget requests,

Program plans

New knowledge,

proof of concepts as

represented by data,

publications

Technology prototypes-initial-intermediate-commercial

Performance analysis

Test, improve, & validate

commercial- scale

technology,Give industry

hands-on experience

Government purchases,Information

disseminated,Early seeding

of technologies

Public & private labs

and test beds,Knowledge

bases,Trained S&T personnel,

Partnerships

Codes and standards,

Trained personnel,

Audits tools,State programs

Concepts & designs with

possible applications,Knowledge spill-over

Investment by industry in

innovative or advanced

commercial products

Favorable policies, capable delivery

channels for EERE

products

Widespread adoption of

EERE products; More productive use

of energy

For

Economic, security, and

environmental benefits

Technology leadership

Programs,CFO, OMB,Congress

Programs,partners

R&Dcommunity

R&D Community, Industry

Relevantindustries

Relevantmarkets

Potentialpurchasers

Potentially commercializ-

able technologies to

replace existing or fill a system need

Political environment

Quality of R&D proposals

Unpredictable nature of R&D

Cost and performance of

competing technologies

Industry willingness to

take risk

Energy prices

State of the economy

Government policies and regulations

ExternalInfluences

Spin-off products and their associated benefits

New products & businesses

Program funding in

appropriate areas;

Efficiency, Fiscal

responsibility

Relevant S&T expertise,

capabilities and facilities to

deliver programs

Feedback Loops

National R&D capabilities, including options if circumstances change

April 4, 2004 11

Logic models help programs identify:

• Measurement areas for which indicators (metrics) may be developed

• Performance targets for each indicator• Evaluation questions

April 4, 2004 12

Each box in the logic model is a potential measurement area

Inputs

Activities

Outputs

Outcomes

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology

Federal funding (millions of nominal $) Federal personnel (FTEs)Private funding (millions of nominal $) # of RD&D facilities


infrastructure



% programs w/benefit estimates

% program w/program

plans

# of journal articles

# of presentations

# prototypes-initial-intermediate-commercial

Prototype cost & performance

# and % of commercial-

scale technologies

validated

# of tech’s purchased by

gov’t,# of materials disseminated,# of website

hits

# of partnerships

# codes and standards,# personnel

trained,# audits,# state

programs

# journal article citations

# of innovative or advanced commercial

products with improved cost &

performance

# of recommenda-tions for using

advanced commercial

products and practices

# and % of advanced

commercial products by

adoption stage

For

Energy saved (quad. Btu), GW of add’l RE capacity,Expenditure savings ($)

Carbon saved (mmtce)


Programs,partners

R&Dcommunity


Relevantindustries

Relevantmarkets

Potentialpurchasers

# of potentially commercializ-

able technologies



(varies by technology)

Oil prices ($/barrel)

NG prices ($/tcf)

Electricity prices (c/kWh)

Coal prices ($/ton)

GDP (billion 1996 $)

RE production tax credit (c/kWh)

EE/RE tax credits ($)

CAFÉ standards

(mpg)

Vehicle & power plant

emission standards(varies by pollutant)

ExternalInfluences

# of technology spinoffs

Feedback Loops

April 4, 2004 13

Inputs

Activities

Outputs

Outcomes

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology



infrastructure




% program w/program

plans


# of presentations




scale technologies

validated



hits

# of partnerships



programs




performance


advanced commercial


# and % of advanced


adoption stage

For




Programs,partners

R&Dcommunity


Relevantindustries

Relevantmarkets

Potentialpurchasers


able technologies





NG prices ($/tcf)


Coal prices ($/ton)




CAFÉ standards

(mpg)



ExternalInfluences


Feedback Loops

Performance targets may also be developed for each box in the logic model

Cost of Hydrogen

($/gallon gasoline equivalent)

2003200

62010

Non-renewables

5.00 3.00 1.50

Renewables 6.20 5.30 3.90

April 4, 2004 14

Arrows between the boxes help identify evaluation questions

Activities

Outputs

Outcomes

For


Test, improve, & validate commercial-

scale technology,Give industry hands-on

experience

Investment by industry in innovative or

advanced commercial products

Relevantindustries

Evaluation Questions:

• Has industry experience lead to investment in these innovative or advanced commercial energy products?

• Have relevant industries gained hands-on experience with the technologies?

• Have EERE’s efforts to test and demonstrate energy technologies led to validated commercial scale technologies? (Activities to outcomes)

• Has EERE engaged the right partners in tests and done it efficiently?

April 4, 2004 15

Potential management / evaluation questions for EERE ask if strategies are working

1. Have program planning and assessment activities increased potential program benefits per federal dollar spent?

2. Has developing and maintaining EERE program infrastructure provided the scientific and technical expertise and facilities required to carryout program activities?

3. Has research conducted by EERE yielded energy-related concepts and designs with possible commercial applications?

April 4, 2004 16

Management / evaluation questions (cont.)

4. Has EERE technology development yielded potentially commercializable energy technologies?

5. Have EERE efforts to test and demonstrate energy technologies led to investment by industry in these innovative or advanced commercial energy products?

6. Have EERE efforts to develop government and business infrastructures led to favorable policies and capable delivery channels for EERE products and practices?

April 4, 2004 17

Management / evaluation questions (cont.)

7. Have EERE deployment activities with end users increased the awareness, appreciation, and adoption of EERE products and practices?

And, altogether has the EERE portfolio of RDD& D programs led to adoption of EERE products and practices and yielded economic, security, and environmental benefits that would not have occurred otherwise?

April 4, 2004 18

Inputs

Activities

Outputs

Outcomes

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology



infrastructure




% program w/program

plans


# of presentations




scale technologies

validated



hits

# of partnerships



programs




performance


advanced commercial


# and % of advanced


adoption stage

For




Programs,partners

R&Dcommunity


Relevantindustries

Relevantmarkets

Potentialpurchasers


able technologies





NG prices ($/tcf)


Coal prices ($/ton)




CAFÉ standards

(mpg)



ExternalInfluences


Feedback Loops

Evaluations can explain why some goals were met and others were not

(1) Funding was about what was expected

(2) R&D yielded expected results

(3) But, competing technologies improved more than expected

(4) Energy prices were lower than expected

(5) Thus, market penetration was slower than expected

(6) And benefits were lower than expected

April 4, 2004 19

In summary, logic models for RDD&D programs help identify…

Inputs

Activities

Outputs

Outcomes

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology



infrastructure




Budget requests,

Program plans

New knowledge,



publications




commercial- scale


hands-on experience



of technologies




Partnerships


Trained personnel,






commercial products


channels for EERE

products



of energy

For





Programs,partners

R&Dcommunity


Relevantindustries

Relevantmarkets

Potentialpurchasers










take risk

Energy prices



ExternalInfluences



Program funding in

appropriate areas;

Efficiency, Fiscal

responsibility



deliver programs

Feedback Loops


Performance Targets

Cost of Hydrogen

($/gallon gasoline equivalent)

2003 2006 2010

Non-renewables

5.00 3.00 1.50

Renewables 6.20 5.30 3.90

Indicators•Cost of Hydrogen ($/gal gasoline equivalent)

–Non-renewables–Renewables

Evaluation Questions• Did development activities lead to

potentially commercializable technologies?

April 4, 2004 20

… and tell the performance story

Inputs

Activities

Outputs

Outcomes

Program planning &

assessment

Conduct research

Developtechnology


Deploy technology



infrastructure




Budget requests,

Program plans

New knowledge,



publications




commercial- scale


hands-on experience



of technologies




Partnerships


Trained personnel,






commercial products


channels for EERE

products



of energy

For





Programs,partners

R&Dcommunity


Relevantindustries

Relevantmarkets

Potentialpurchasers










take risk

Energy prices



ExternalInfluences



Program funding in

appropriate areas;

Efficiency, Fiscal

responsibility



deliver programs

Feedback Loops


April 4, 2004 21

For more information contact:

Gretchen Jordan

Sandia National Laboratories

950 L'Enfant Plaza, SW

Suite 110

Washington, DC 20024-2123

Phone: (202) 314-3040

Email: [email protected]

John Mortensen: [email protected]

John Reed: [email protected]

George Teather: [email protected]

mailto:[email protected]




Using Logic Models in Managing Performance of Research and Technology Programs: An example for a...

Documents

Transcript of Using Logic Models in Managing Performance of Research and Technology Programs: An example for a...