Attributing Benefits to Voluntary Programs: Practical and Defensible Approaches

IEc

INDUSTRIAL ECONOMICS, INCORPORATED

Attributing Benefits to Voluntary Programs: Practical and Defensible Approaches

Cynthia Manson, Principal

June 23, 2011

INDUSTRIAL ECONOMICS, INCORPORATED 2

Project History

EPA ORCR (OSW) faced OMB concerns:•Economic benefits of partnership programs•Specific ICRs – WasteWise and NPEP•Economic efficiency of programs (PART)

Identified need to:•Respond to demand for robust analysis

• Noting data limitations of partnership programs• Programs already exist, limits analytic options

•Harmonize discussions of economic analysis and program evaluation

Result:•Framework for analysis using available data•Discussion of limitations of experimental design


Economic reasoning for voluntary programs

• To address market failures:• Imperfect information in the

marketplaceSIGNALING FAILURE

• Lack of knowledge transfer on green approaches from firm to firm“PUBLIC GOOD” NATURE OF R&D

• To address unregulated or under-regulated areas, e.g., water conservation, pollution prevention


Potential Impacts of EPA Partnership Programs

Example Programs: WasteWise, EnergyStar, Natural Gas Star, WaterSense, Green Suppliers Network

Provide incentives for participants to share and adopt greener behaviors that, in absence of EPA assistance, would have occurred:

• Later in time• On a temporary or tenuous basis• On a smaller scale• Not at all


Potential Impacts of EPA Partnership Programs

• “Technical Assistance:” Goal of Information sharing – transfer of R&D, innovation. EPA facilitates transfer of innovations among participants, and to non-participants through web sites and publications.• Addresses “public good nature of R&D”• Spillover effects DELIBERATE

• Market signaling: EPA recognition informs consumers about environmental quality, though:• Awards and other public recognition;• Logos that signal participation and performance;• Certification assistance and verification; and• Marketing assistance.• Addresses “signaling failure”


Problem: “Proving” program outcomes• Optimal design: randomized control trial

(RCT)• Strongest approach - addresses causality,

attributes program benefits.• Requires random assignment of groups to

participate and not (drug tests).

• Random assignment not possible in most EPA contexts, including voluntary programs

• Alternative to RCT: two-stage approach:• Evaluate features of participant group, ensure

appropriate selection of control group(s). • Approach still requires identifying non-

participants.

• Spillover deliberate – no control group.


Proposed Approach: Tiered Assessment with existing data

• Level 1: Threshold Assessment: ensures and documents that the program design is appropriate for addressing market failure.

• Level 2: Intervention-Outcome Assessment: verifies that program resources and activities are logically aligned with desired outcomes.

• Level 3: Quasi-Experimental Design: Quantitative analyses that effectively attribute benefits to the program, while avoiding feasibility issues of experimental design.


Level 1: Threshold Assessment: Technical Assistance

Threshold evidence for potential technical assistance benefits of a partnership program - innovations are:

• Non-patentable• Applicable broadly to other firms• Able to be duplicated by other firms at low cost• Able to be duplicated by other firms quickly• Applicable to small firms with numerous competitors


Level 1: Threshold Assessment: Market signaling

Threshold evidence for potential market signaling benefits to a partnership program:

• Environmental quality characteristics are difficult for the public to observe

• Environmental quality characteristics are not already addressed by a respected third- party certification of auditing scheme


Level 2: Intervention-Outcome Assessment:

Thorough inventory of program interventions and outcomes (quantified logic model).

Step 1: Information on interventions should include:


Level 2: Intervention-Outcome Assessment:

Step 2: Information on outcomes should include:


Level 2: Intervention-Outcome Assessment: “Logic Model” Example


Level 3: Quasi-Experimental DesignExamples of quasi-experimental designs:

• Sub-optimal comparison group: Compare participants and non-participants without statistical correction.

• Regression discontinuity: Assign participants to a treatment or comparison group on the sole basis of a cutoff score on a pre-program measure.

• Time series: Measure indicators of study group performance over time, with or without a comparison group.

• Outcome analysis: Measure changes in outcome variable(s) without accounting for external factors.


Illustration of Boyd’s Tiered Approach

IEcINDUSTRIAL ECONOMICS, INCORPORATED

617.354.0074

Attributing Benefits to Voluntary Programs: Practical and Defensible Approaches

Documents

Transcript of Attributing Benefits to Voluntary Programs: Practical and Defensible Approaches