Post on 30-May-2018
8/14/2019 Tom Koontz Friday
1/18
11
Evaluating the Performance of
Collaborative Environmental
Governance
Tomas M. KoontzTomas M. Koontz
School of Environment & Natural ResourcesSchool of Environment & Natural Resources
Ohio State UniversityOhio State University
Craig W. ThomasCraig W. Thomas
Daniel J. Evans School of Public AffairsDaniel J. Evans School of Public Affairs
University of WashingtonUniversity of Washington
8/14/2019 Tom Koontz Friday
2/18
22
The Problem We FaceThe Problem We Face
Collaboration has increasingly replaced otherCollaboration has increasingly replaced otherforms of environmental governanceforms of environmental governance
Such as centralized planning andSuch as centralized planning and
regulationregulation This may be a good thingThis may be a good thing
But we have very little knowledge about theBut we have very little knowledge about the
environmental outcomes of collaborationenvironmental outcomes of collaboration Collaboration appears to be good for socialCollaboration appears to be good for social
outcomes (like social capital)outcomes (like social capital)
But it may be bad for the environmentBut it may be bad for the environment
8/14/2019 Tom Koontz Friday
3/18
33
What is Collaborative Governance?What is Collaborative Governance?
Multiple governmental andMultiple governmental and
nongovernmental stakeholdersnongovernmental stakeholders
working together to develop andworking together to develop and
advance a shared visionadvance a shared vision
8/14/2019 Tom Koontz Friday
4/18
44
Research QuestionsResearch Questions
Is collaborative governance good for theIs collaborative governance good for the
environment?environment?
Does collaborative governance produceDoes collaborative governance producebetter environmental outcomes than otherbetter environmental outcomes than other
forms of governance (such as centralizedforms of governance (such as centralized
planning or regulation)?planning or regulation)?
8/14/2019 Tom Koontz Friday
5/18
55
How Collaboration Might AffectHow Collaboration Might Affect
the Environmentthe Environment
Negatively: Lower common-denominatorNegatively: Lower common-denominator
solutions during policy adoptionsolutions during policy adoption
Positively: More durable solutions duringPositively: More durable solutions during
implementationimplementation
8/14/2019 Tom Koontz Friday
6/18
66
Inputs
(e.g., human,
financial,
and technical
resources)
End
outcomes
(social and
environmental)
Processes
(e.g., collabor-
ation)
Intermediate
outputs
(e.g., plans)Intermediate
outcomes
(e.g. human
behavior)
All other
non-programmatic
variables
End
outputs
(e.g., projects)
Basic Logic Model
8/14/2019 Tom Koontz Friday
7/18
77
State of the Literature onState of the Literature on
Collaborative EnvironmentalCollaborative Environmental
GovernanceGovernance
Empirical research has focused on:Empirical research has focused on:
Inputs (e.g., human, technical, financial)Inputs (e.g., human, technical, financial) Processes (e.g., consensus, ADR)Processes (e.g., consensus, ADR)
Outputs (e.g., plans, permits, projects)Outputs (e.g., plans, permits, projects)
Social outcomes (e.g., trust, social capital)Social outcomes (e.g., trust, social capital)
Very little research on environmental outcomesVery little research on environmental outcomes
8/14/2019 Tom Koontz Friday
8/18
88
Recent Empirical Claims aboutRecent Empirical Claims about
Environmental OutcomesEnvironmental OutcomesLeach and Sabatier (2005) measureLeach and Sabatier (2005) measure
participantparticipant perceptionsperceptions of environmentalof environmental
changechange
Others use outputs as proxies for outcomesOthers use outputs as proxies for outcomes
(either intentionally or carelessly)(either intentionally or carelessly)- Salmon recovery plans; farmland preservation;- Salmon recovery plans; farmland preservation;
Superfund projects; choicesSuperfund projects; choices
- Beware: outputs may not be good proxies for- Beware: outputs may not be good proxies for
outcomes, due to intervening variablesoutcomes, due to intervening variables
8/14/2019 Tom Koontz Friday
9/18
99
Possible Research DesignsPossible Research Designs
ExperimentalExperimental
Quasi-experimentalQuasi-experimental
StatisticalStatistical
Case StudiesCase Studies
8/14/2019 Tom Koontz Friday
10/18
1010
Experimental MethodsExperimental Methods
WhatWhat
Randomized trialsRandomized trials
With focus on collaboration as aWith focus on collaboration as a
management interventionmanagement intervention
FeasibilityFeasibility
Very lowVery low Primarily due to long time horizons forPrimarily due to long time horizons for
environmental changeenvironmental change
8/14/2019 Tom Koontz Friday
11/18
1111
Quasi-Experimental MethodsQuasi-Experimental Methods
WhatWhat
Also focus on collaboration as a managementAlso focus on collaboration as a management
interventionintervention
But w/o randomized controls or other featuresBut w/o randomized controls or other featuresof classic experimental designof classic experimental design
FeasibilityFeasibility
Easy to find natural experimentsEasy to find natural experiments
Harder to find matched-case controls (usingHarder to find matched-case controls (using
most-similar comparative methods)most-similar comparative methods)
Harder to find valid longitudinal dataHarder to find valid longitudinal data
8/14/2019 Tom Koontz Friday
12/18
1212
Statistical MethodsStatistical Methods
WhatWhat Analyzes correlations among variablesAnalyzes correlations among variables
to test probability of causal effects (not causalto test probability of causal effects (not causal
mechanisms)mechanisms)
FeasibilityFeasibility
High for existing data setsHigh for existing data sets e.g., PART (with caveat)e.g., PART (with caveat)
Harder to find valid longitudinal dataHarder to find valid longitudinal data
8/14/2019 Tom Koontz Friday
13/18
1313
Inputs
(e.g., human,
financial,
and technical
resources)
End
outcomes
(social andenvironmental)
Processes
(e.g., collabor-
ation)
Intermediate
outputs
(e.g., plans)Intermediate
outcomes
(e.g. human
behavior)
All other non-programmatic
(exogenous) variables
End
outputs
(e.g., projects)
Experimental, Quasi-Experimental,and Statistical Methods
8/14/2019 Tom Koontz Friday
14/18
1414
Case Study MethodsCase Study Methods
WhatWhat Emphasizes process tracing in causalEmphasizes process tracing in causal
sequencessequences
Searches for causal mechanisms, not justSearches for causal mechanisms, not justcausal effectscausal effects
FeasibilityFeasibility
HighHigh But requires careful use of counterfactuals,But requires careful use of counterfactuals,
congruence with theory, and controlledcongruence with theory, and controlledcomparisons (holding other variables constant)comparisons (holding other variables constant)
8/14/2019 Tom Koontz Friday
15/18
1515
Inputs
(e.g., human,
financial,
and technical
resources)
End
outcomes
(social andenvironmental)
Processes
(e.g., collabor-
ation)
Intermediate
outputs
(e.g., plans)Intermediate
outcomes
(e.g. human
behavior)
All other non-programmatic
(exogenous) variables
End
outputs
(e.g., projects)
Case Studies
8/14/2019 Tom Koontz Friday
16/18
1616
Accountability for PerformanceAccountability for Performance
Collaboration should be held to the sameCollaboration should be held to the same
performance standards as other forms ofperformance standards as other forms of
governancegovernance
Collaboration may be quite complementaryCollaboration may be quite complementaryto developing performance mgt systemsto developing performance mgt systems
Incentives for partners to monitor, gatherIncentives for partners to monitor, gather
information, develop metrics for successinformation, develop metrics for success
But how can we do it from a methodologicalBut how can we do it from a methodological
perspective?perspective?
8/14/2019 Tom Koontz Friday
17/18
1717
Our Research AgendaOur Research Agenda
In processIn process Case studies of public sector collaborative effortsCase studies of public sector collaborative efforts
To show links from processes through to outcomesTo show links from processes through to outcomes
Coding PART reportsCoding PART reports
Matched case comparisons for statistical analysis:Matched case comparisons for statistical analysis:
collaborative v. non-collaborative cases within programscollaborative v. non-collaborative cases within programs
But many outputs/outcomes are miscodedBut many outputs/outcomes are miscoded
Link to theory for generalizabilityLink to theory for generalizability
Future ResearchFuture Research Quasi-experimental methodQuasi-experimental method
Using remote sensing data, water quality reports, etc.Using remote sensing data, water quality reports, etc.
Ideas for comparative cases, anyone?Ideas for comparative cases, anyone?
How do perf. mgt. systems get started?How do perf. mgt. systems get started?
8/14/2019 Tom Koontz Friday
18/18
1818
ConclusionConclusion Is collaborative governance good for theIs collaborative governance good for the
environment?environment?
Does collaborative governance produceDoes collaborative governance producebetter environmental outcomes than otherbetter environmental outcomes than other
forms of governance (such as centralizedforms of governance (such as centralizedplanning or regulation)?planning or regulation)?
These are very important (and too oftenThese are very important (and too often
unasked) questionsunasked) questions
But they are difficult to answerBut they are difficult to answer
And they will keep us busy for a long timeAnd they will keep us busy for a long time