Environmental Policy Evaluation
-
Upload
wade-mcintosh -
Category
Documents
-
view
42 -
download
0
description
Transcript of Environmental Policy Evaluation
![Page 1: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/1.jpg)
Environmental Policy Evaluation
MADE Program
30/10/2009
![Page 2: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/2.jpg)
Policy Evaluation
• Oversight agencies increasingly concerned about effectiveness of policies (both mandatory and voluntary)
• Voluntary programs/International Agreements (ISO 9001/Kyoto Protocol) are greatly increasing in number and size
• Need for rigorous, scientfic evaluations of policies
![Page 3: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/3.jpg)
Treatment
• Policy is known as the treatment
• Observations of unaffected entities is called the control
• An effective policy will alter the outcome of those treated (given the policy) relative to the control
![Page 4: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/4.jpg)
Policy Story
• It is worthwhile to learn the policy background before starting an evaluation
• Sometimes the proper outcome variable to measure is not clear
• Be willing to listen to those involved in the policy but remember their interests are different than yours – No Stockholm Syndrome
![Page 5: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/5.jpg)
Ideal Story
• You find data that:– Starts before policy went into affect– Continues after policy went into affect– Covers those affected by policy and those
unaffected
• The policy– Randomly assigned those involved– Staggered its starting date
![Page 6: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/6.jpg)
Issue to Control for
• The big issues to control for are:– Pre-policy behavior/trend– Behavior of those unaffected by policy– Selection of those affected by policy– Other variables that affect outcome
• Ideally the policy will have a staggered start so that the analysis is using both time and space variation to determine policies affect
![Page 7: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/7.jpg)
Panel Data
• Usually evaluations occur using panel data methods
• Panel data is when you follow the same entity over time
• Methods often used are fixed effects, random effects, dynamic panel, logit
![Page 8: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/8.jpg)
Voluntary Programs
• Generally between government agency and a group of firms
• Firms join the program (become a partner) and pledge to make specified environmental improvement
• Partners potentially receive benefit in form of confidential information
![Page 9: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/9.jpg)
Standard Analysis
• Conventional interpretation of partnership programs is that information given to partners bring exclusive benefits
• Selection bias likely to occur; needs to be controlled for
• Typical View - Program effective if partners had statistically better environmental outcome than non-partners
![Page 10: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/10.jpg)
Estimation• Difference in Difference Model
– Post Policy Dummies (T)– Partner Dummy (P)– Partner & Policy Dummy Interaction (TP)– Other controls that alter dependent variable
(X, S, N)
• β9 is the relevant variable
itiititititjtitit vPTTPNSXR ^
987654
![Page 11: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/11.jpg)
Diff-in-Diff Variable
• β9 tells us whether the group affected by the policy acted differently than the group unaffected, once the policy went into affect
• If it is positive & statistically significant, the policy is associated with an increase in dependent variable
• Vice Versa for negative coefficients
![Page 12: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/12.jpg)
Other Interpretations
• Policy variable (β8) tells us whether the dependent variable changed when policy went into affect
• Partner variable (β7) tells us whether partners are inherently different with respect to dependent variable
![Page 13: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/13.jpg)
13
Example: C2P2 program• Coal Combustion Partnership Program (C2P2) started
in 2001, part of Resource Conservation Challenge– promotes beneficial uses of Coal Combustion Products (CCP)
as opposed to lanfilling them– CCP can be reused in concrete, asphalt, & more
• About 170 Partners– Work with industry groups & other federal agencies to promote
education about CCP reuse– Outreach/networking/acknowledgement for CCP suppliers and
demanders – Document reuse by individual facilities, companies and
government organizations
![Page 14: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/14.jpg)
Question• Is C2P2 associated with an increase in CCP
re-use?
• Data on coal power plants in US for 96-05 (C2P2 started in 2001)
• Dependent Variable: Fly Ash Re-Use Rates – the amount of fly-ash sold divided by the total
fly-ash generated
![Page 15: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/15.jpg)
15
Rate of Fly Ash Reuse
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year
Re-
Use
Rat
io
Initial Partners
Non-Partners
Late Partners
![Page 16: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/16.jpg)
Raw Data
• Be careful with Raw Data like this
• Other co-variates not controlled for
• Policy makers tend to like these pictures even though they are not telling the whole story
![Page 17: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/17.jpg)
17
Estimation• Random Effects Model
• Dependent Variable:– Ratio of Fly Ash Reused to Total Fly Ash Generated
• Controls:– Pre, Early, & Late C2P2 Time Dummies – Partner Variable & C2P2 Time Interactions– Total Coal Burnt– Ash Content of Coal– Price of Aggregates/Cement in State– Level of Cement Imports– Region Dummies– NOx Control Equipment Present
itiititititjtitit vPTTPNSXR ^
987654
![Page 18: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/18.jpg)
18
Evaluation ResultsTime Period : 1996-2005Dependent Variable: Model: Fixed Effects Random EffectsExplanatory Variable Coefficient Stnd. Error Coefficient Stnd. ErrorPredicted Partner -0.18** 0,08Early-C2P2 (2001-2002) 0.06** 0,02 0.08** 0,03Late-C2P2 (2003-2005) 0.06** 0,02 0.07** 0,02Early-C2P2* Predicted Partner 0,03 0,02 0,03 0,02Late C2P2* Predicted Partner -0.01 0,02 -0.01 0,02Number of Utility's Plants Near 0.01* 0 0,01 0,01Post-C2P2* Utility's Plants Near 0,01 0,01 0,01 0,01Lag Utility Reuse 0.08** 0,04 0.11*** 0,03Aggregates Price -0.01 0,02 0.06*** 0,02Cement Price 0,01 0,01 -0,01 0,01Cement Imports 0,01 0,01 0,01 1SCR Installed 0,01 0,02 0,01 0,01Avearge Ash Content -0.01*** 0 -0.02*** 0,01Total Coal 0.01** 0,00 0.01*** 0,00N 1883 1883Plants 303 303R2 0,04 0,3Region & Year Dummies Not Shown for BrevityErrors Clustered by Utility for all specifications and Bootstrapped for the first two specifications*, **, *** indicate 10%, 5% and 1% statistical significance, respectively
Fly Ash Reuse RatioFly Ash Reuse Ratio
![Page 19: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/19.jpg)
Other Issues
• Treatment Spillovers
• Selection Bias
• Error Specification
![Page 20: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/20.jpg)
20
Treatment Spillovers• Programs provide information that spills-over
to non-partners; programs may be public good in nature
– Technical information-spreading is not limited to partners, thus both partners and non-partners can take advantage of program
– One country agreeing to reduce consumption of a good could lead to increases in price for other countries
![Page 21: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/21.jpg)
Evaluation Interpretation
• If there are treatment spillovers,– Partners and non-partners do not need to be
statistically different for program to be effective– Requires evidence that spillovers exist– Requires statistically better environmental
outcomes for both groups after program initiation
![Page 22: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/22.jpg)
22
Evidence of Spillovers: C2P2
• Non-partner data analyzed for evidence of treatment spillover by comparing non-partners in states w/ many partners to non-partners in states w/ few partners
![Page 23: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/23.jpg)
Example: C2P2Time Period : 1996-2005Dependent Variable: Model: Fixed Effects Random EffectsExplanatory Variable Coefficient Stnd. Error Coefficient Stnd. ErrorModerate/Low C2P2 Partners Nearby 0.34*** 0,08Early-C2P2 (2001-2002) 0.10** 0,04 0.11** 0,04Late-C2P2 (2003-2005) 0.14** 0,05 0.15** 0,05Early-C2P2* Moderate/Low C2P2 Partner -0.09* 0,05 -0.09* 0,05Late-C2P2* Moderate/Low C2P2 Partner -0.14** 0,06 -0.14** 0,06Number of Utility's Plants Near -0.01 0,01 -0.01 0,01Post-C2P2* Utility's Plants Near 0,01 0,02 0,01 0,02Lag Utility Reuse 0,07 0,06 0,04 0,06Aggregates Price -0.01 0,02 -0.01 0,02Cement Price -0.01 0,01 -0.01* 0,00Cement Imports 0,01 0,01 0,01 0,01SCR Installed 0,04 0,03 0,04 0,04Avearge Ash Content -0.01** 0 -0.02** 0,01Total Coal -0.01 0,01 0.01** 0,00N 1088 1088Plants 176 176R2 0,2 0,48State & Year Dummies Not Shown for BrevityErrors Clustered by Utility*, **, *** indicate 10%, 5% and 1% statistical significance, respectively
Fly Ash Reuse Ratio Fly Ash Reuse Ratio
![Page 24: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/24.jpg)
Selection Bias
• If entities are not randomly assigned to treatment or control, selection bias could be a problem
• If treatment is mandatory, selection bias is less likely to be a problem
• If treatment is voluntary, then selection bias is likely to be a problem
![Page 25: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/25.jpg)
Endogeneity
• Another way to describe selection bias is to say that the choice of enetering the voluntary progam is endogenous
• This generally means that the choice of entering the program affects the outcome measure, rather than the outcome measure affecting the choice of entering the program.
![Page 26: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/26.jpg)
Direction of Bias
• One big problem is that the direction of bias could be either positive or negative
• If entities join program because it is easy to acheive goals set, the bias is positive
• If entities join program because they really need the help, the bias is negative.
![Page 27: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/27.jpg)
Example-Positive
• Norway joins the Oslo Protocol because it has large natural gas reserve, making reducing sulfur emissions relatively cheap
• If evaluate the effect of the Protocol on Norway (relative to Poland for example), it will look like the Protocol had a large effect when in reality the change in behavior is driven by the switch to natural gas.
![Page 28: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/28.jpg)
Example-Negative• Poland joins the Sophia Protocol even
though it relies heavily on coal for electricity production
• If we evaluate the effect of the Protocol on Poland (relative to Sweden for example) it will look like the Protocol is having no effect when in reality it is the lack of alternatives for electricity that is driving the result.
![Page 29: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/29.jpg)
Correcting for Endogeneity
• Generally known at Instrumental Variables
• Find other variables that predict the variable that is endogenous
• In the case of selection bias into the policy treatment, this generally means variables that are associated with the benefits or costs of recieveing the treatment
![Page 30: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/30.jpg)
30
Partner Choice Example: C2P2• Dependent Variable:
– Partner Designation
• Controls:– Investment in Solid Waste Disposal 96-00– State w/ CCP re-use authorized– State w/o CCP Permit Requirement– Utility Size– Avg Fly and Bottom Ash Reuse 96-00– Avg Price of Agg. & Cement in State 96-00
![Page 31: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/31.jpg)
31
Example : C2P2Dependent Variable: Model: ProbitExplanatory Variable Coefficient Stnd. ErrorAvg Fly Ash Reuse Ratio 1996-2000 -0.11 0.28Avg Bottom Ash Reuse Ratio 1996-2000 0.2 0.24Avg Aggregates Price 1996-2000 0.34*** 0.08Avg Cement Price 1996-2000 -0.01 0.01Utility Size 0.04*** 0.01Restructed Market -0.41* 0.22NOx SIP Call State 0.15 0.5Total Solid Waste Disposal Investment 1996-2000
0.03 0.02
No CCP Permit Required 0.45** 0.21Re-use Guidelines Set 1.1*** 0.3N 305R2 0.18*, **, *** indicate 10%, 5% and 1% statistical significance
Initial Partner Dummy
![Page 32: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/32.jpg)
Error Specification
• Generally in panel methods, you would like to cluster errors by largest grouping entity possible
• These could be industry classification, firm (as opposed to plant), country, etc
• This controls for unobserved correlation better than a narrow grouping
![Page 33: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/33.jpg)
Other Options
• Bootstrapping of error terms is a catch all – Code is quite easy in Stata to code
• Many other potential fixes available, Stata help very good for this
![Page 34: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/34.jpg)
Policy Evaluation
• Important to:– Consider pre-policy trends– Consider control group’s actions– Other co-variates– How entities enter the policy– How the treatment affects entities
• With these issues covered, you have done as much as is possible as a data user
![Page 35: Environmental Policy Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062308/56812e3c550346895d93b4c4/html5/thumbnails/35.jpg)
Questions?
• About Data
• About Methods
• About Examples
• About Information on Policies
• About dealing with policymakers
• …