Measuring Public Policy - NCSL
Transcript of Measuring Public Policy - NCSL
Measuring Measuring Public Public Policy Policy
John Turcotte
DirectorProgram Evaluation Division
North Carolina General Assembly
National Association of Legislative Fiscal Officers
Portland, Maine
September 6, 2007
22
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Presentation OverviewPresentation OverviewImportance of insisting that agency executives focus on what the public wants--not so much on how much money and effort goes into programsHow to decode “accountababel” by differentiating between planning, organization, system, and relative quality termsGetting better outcome measures by using the “So What?” testGetting better utility or “scalability” from unit cost measuresKey questions about performance measures as a whole
33
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Public Policy Debate Over Means Often Public Policy Debate Over Means Often Clouds the EndsClouds the Ends
How will we know that we have what we want?
Whatdo we want?
What do we need to do to get what we want?
What resources are necessary?
Who will supply the resources?
In order to comply with state law and to accomplish our mission, we need continued financial support.
We believe we do a good job and could do more if we only had more resources. The public demands this.
Inflation is sapping our ability to serve the people.
Our employees have gone two budget cycles without raises.
Do you have any questions?
In order to comply with state law and to accomplish our mission, we need continued financial support.
We believe we do a good job and could do more if we only had more resources. The public demands this.
Inflation is sapping our ability to serve the people.
Our employees have gone two budget cycles without raises.
Do you have any questions?
44
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Accountability is Improved by a Accountability is Improved by a Right to Left ViewRight to Left View
How will we know that we have what we want?
Whatdo we want?
What dowe need to do to get what we want?
What resources are necessary?
Who will supply the resources?
55
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
AccountababelAccountababel——A Confusing Blend A Confusing Blend ofof Organizational, Planning, Systems Organizational, Planning, Systems and Relative Quality Terms and Relative Quality Terms
Last year, the enforcement program of the Department of Justice adhered to our mission, achieved goals, met objectives, improved services and activities and were
efficient, effective, and economical. We stayed on task, increased outputs and achieved outcomes while
holding inputs constant.
66
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Accountability Oversight: Accountability Oversight: How to DeHow to De--Code the Terms Code the Terms Used in the “Discipline”Used in the “Discipline”
PlanningPlanning Terms: Used for setting a Terms: Used for setting a program in motion and measuring progressprogram in motion and measuring progressOrganizationOrganization Terms: Used for sorting out Terms: Used for sorting out where work is donewhere work is doneSystemSystem Terms: Used for modeling the logic Terms: Used for modeling the logic of all parts of a programof all parts of a programRelative QualityRelative Quality Terms: Used for analysis Terms: Used for analysis of alternativesof alternatives
77
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Planning TermsPlanning Terms
PremisePremiseGoalsGoalsObjectivesObjectivesTasksTasks
Free Market
• Compelling reason for governmental intervention because the market is not working
• Intervention consists of a scheme that is supposed to produce results
We spend entirely too much time on program and not nearly enough on premise.
Former US Senator Daniel Patrick Moynihan, N.Y.
PremisePremise
Premise
Mission
Goals
Objectives
Tasks
Objectives and Tasks Should be Measurable:
1. Active Verb (To arrest)
2. Quantifier (90%)
3. Object (of suspects)
4. Time Frame (within 72 hours after arrest warrant is issued)
Tasks are components of and action steps required to accomplish an objective
Planning TermsPlanning Terms
System TermsSystem Terms
Source: Adaptation by Turcotte of W.K. Kellogg Foundation: Using Logic Models to Bring Together Planning, Evaluation, and Action:Logic Model Development Guide, 2004, p. 1.
Planned Work Intended Result
Performance MEASURES
Performance INDICATORS
Financial Audit Performance Audit Program Evaluation
1212
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
What Should Outcome What Should Outcome Measures Measure?Measures Measure?
Rates of occurrence compared to previous year, to a “stretch” standard previously set, and if possible to comparable jurisdictions
Negatives: inmate idleness, escapes, assaults on fellow inmates and staff, repeat offenders, substance abuse during incarceration, inmate disease and debilitating conditionsPositives: inmate completion of education programs, post incarceration employment and earnings, victim repayment
1313
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Use the Use the So WhatSo What? Test to Differentiate ? Test to Differentiate Outputs from Outcomes and Improve Outputs from Outcomes and Improve
Outcome MeasuresOutcome Measures
We examined 6,800 or 94% of inmates for substance abuse in FY 2005. “So what?”We discovered that 20% of them were substance abusers. “So what?”We provided 100% of them with treatment regimens including counseling and appropriate therapies. “So what?”64% of the inmates we have treated to date have completed the program. “So what?”70% of completers were tested substance free after 60 days. “So what?”50% of completers were tested substance free after three years, and 70% of those remained substance free during parole and 90% of these were substance free after two years after completing parole. Our completion and substance-free rates were the highest among all US prison populations. “That’s better.”
1414
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Relative Quality TermsRelative Quality Terms
Economical
EfficientEffective
1515
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
EfficientEfficientThe degree to which a system or The degree to which a system or
component performs its designated component performs its designated functions with minimum consumption functions with minimum consumption of resourcesof resources .. Deploying the minimum Deploying the minimum number of guards needed to secure a number of guards needed to secure a corrections facility.corrections facility.
1616
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
EffectiveEffectiveAchieves Achieves
resultsresults. E.g. . E.g. provides provides highest level of highest level of security to security to prevent prevent escapes and escapes and assaults on assaults on inmates and inmates and guards. guards.
"Supermax" Penitentiary in Florence, Colorado, (Photo: DLR Group)
1717
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
EconomicalEconomicalA lower cost
alternative. A house arrest program cost less per inmate than a minimum security facility.
1818
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
The Holy Grail: Cost The Holy Grail: Cost EffectivenessEffectiveness
Economical
EfficientEffective
1919
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
ScalabilityScalability is very important!is very important!Could the Legislature build a Could the Legislature build a totaltotal
budget for an activity by multiplying budget for an activity by multiplying each unit cost (cost per output) by each unit cost (cost per output) by the number of expected outputs?the number of expected outputs?
If not, the “unit cost” number is If not, the “unit cost” number is inaccurate and of little use.inaccurate and of little use.
Ditto for time duration per output. Ditto for time duration per output. Should be scalable for computing Should be scalable for computing staff positions needed.staff positions needed.
Key Issue: Scalability of Unit Key Issue: Scalability of Unit Cost and Time Duration Cost and Time Duration
Efficiency MeasuresEfficiency Measures
2020
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
Key Questions About Key Questions About Performance Performance Measures as a Whole Measures as a Whole
Is the logic model and premise clear and understood?Is the logic model and premise clear and understood?Are measures unequivocal? (No terms such as most, sufficient) Are measures unequivocal? (No terms such as most, sufficient) Are supporting numbers accurate and verifiable?Are supporting numbers accurate and verifiable?Are measures properly classified? (Output or Outcome)Are measures properly classified? (Output or Outcome)Are measures Are measures comprehensivecomprehensive? Are all important or more costly ? Are all important or more costly activities measured?activities measured?Are measures properly constructed? (Appropriate metrics)Are measures properly constructed? (Appropriate metrics)Are there Are there quality quality standards for outputs and outcomes?standards for outputs and outcomes?What are the expected What are the expected quantityquantity standards for outputs and outcomes? standards for outputs and outcomes? Are these a “stretch”?Are these a “stretch”?Are measures portrayed graphically or in easyAre measures portrayed graphically or in easy--toto--use format?use format?
2121
John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007
John TurcotteJohn TurcotteDirectorDirector
North Carolina General AssemblyNorth Carolina General AssemblyProgram Evaluation DivisionProgram Evaluation Division
[email protected]@ncleg.net