Module 8 Monitoring Observed Outcomes

8/8/2019 Module 8 Monitoring Observed Outcomes

http://slidepdf.com/reader/full/module-8-monitoring-observed-outcomes 1/15

Module 8 Module 8

Monitoring ObservedMonitoring ObservedPolicy OutcomesPolicy Outcomes



Social systems Lorenz curve

accounting Gini Index

Regression discontinuity Purchasing power Social experimentation Random innovation

Social auditing Quasi-experimentation

Research and practice Evaluability assessment

synthesis Internal validity

Threats to validity External validity

Interrupted time-series Control-seriesCurrent Euros Constant Euros

Key Terms and Concepts



POLICYPERFORMANCE

EXPECTEDOUTCOMES

PREFERREDPOLICIES

OBSERVEDOUTCOMES

ForecastingEvaluation

Monitoring Recommendation

problem

structuring

problem

structuring

POLICYPROBLEMS



The Importance of TimeThe Importance of TimeRETROSPECTIVE:What happened? Wasit worthwhile?

PROSPECTIVE:What will happen?Will it be worthwhile?

OBSERVEDOUTCOMES

EXPECTEDOUTCOMES

Monitoring

POLICIES

Forecasting



Why Monitoring Is ImportantWhy Monitoring Is Important

It is not that we hav e so many It is not that we hav e so many

well well- -designed policies. R ather, wedesigned policies. R ather, we

hav e more well hav e more well- -designed policiesdesigned policiesthan we hav e w ays t o moni t or than we hav e w ays t o moni t or

them. Wi thou t moni t oring, wethem. Wi thou t moni t oring, we

c annot know a good policy from a c annot know a good policy from a bad onebad one²²or w hether the policy is a or w hether the policy is a

policy at all.policy at all.



An Unmonitored Policy MayAn Unmonitored Policy May

Conceal A Disabled VehicleConceal A Disabled Vehicle

Failing to monitor Failing to monitor the outcomes of athe outcomes of apolicy is likepolicy is like

counting thecounting theamount of amount of gasoline a car gasoline a car has consumedhas consumedwithout seeingwithout seeinghow far it hashow far it hastraveled.traveled.



Four Functions of MonitoringFour Functions of Monitoring

C ompli anceC ompli ance²²Are laws on local governmentAre laws on local government

consistent with EU requirements?consistent with EU requirements?

Audi t ing Audi t ing ²²Are revenues intended for localAre revenues intended for local

communities reaching them?communities reaching them?

Account ing Account ing ²²Are policies on educationalAre policies on educational

reform producing qualified students?reform producing qualified students?

Ex pl anat ionEx pl anat ion²²Are outcomes of a policyAre outcomes of a policycaused by the policy, or by other factors?caused by the policy, or by other factors?



Approaches to MonitoringApproaches to Monitoring

Social Systems AccountingSocial Systems Accounting

Social AuditingSocial Auditing

Research and Practice SynthesisResearch and Practice Synthesis

Policy ExperimentationPolicy Experimentation



Social Systems AccountingSocial Systems Accounting

HousingHousing²²Area per person (square meters)Area per person (square meters)

Average Life ExpectancyAverage Life Expectancy

Quality Adjusted Life YearsQuality Adjusted Life Years

Income Distribution (Gini Index)Income Distribution (Gini Index)

Air Pollution Index (parts per million)Air Pollution Index (parts per million)

Lead Concentration Index (bloodLead Concentration Index (blood

concentration)concentration) Persons in Mental HospitalsPersons in Mental Hospitals

Persons Below Poverty LinePersons Below Poverty Line



Social Auditing With User SurveysSocial Auditing With User Surveys

PolicyPolicy--Program SpecificationProgram Specification²²What goals,What goals,objectives, and resources constitute theobjectives, and resources constitute thepolicy?policy?

Collection of Available InformationCollection of Available Information²²WhatWhatinformation is available on inputs, processes,information is available on inputs, processes,

outputs and impacts?outputs and impacts? Policy ModelingPolicy Modeling²²What causal mechanismsWhat causal mechanisms

link inputs and processes to outputs andlink inputs and processes to outputs andimpacts?impacts?

EvaluabilityEvaluability AssessmentAssessment²²

Is the policy clear Is the policy clear enough and unambiguous to know what toenough and unambiguous to know what tomonitor?monitor?

Collection of New InformationCollection of New Information²²What newWhat newinformation needs to be collected?information needs to be collected?



Research and Practice SynthesisResearch and Practice Synthesis

Synthesis of research on planned change,Synthesis of research on planned change,

communication of innovations, socialcommunication of innovations, social

marketing strategies (journals and books)marketing strategies (journals and books)

Synthesis of published and unpublishedSynthesis of published and unpublishedpolicy documents (memos, reports,policy documents (memos, reports,

statistics)statistics)

Synthesis of reported cases of change,S

ynthesis of reported cases of change,innovation, and reform (case surveyinnovation, and reform (case survey

analysis)analysis)



Threats to Validity (Rival Hypotheses)Threats to Validity (Rival Hypotheses)

When Conducting Policy ExperimentsWhen Conducting Policy Experiments

Statistical Conclusion ValidityStatistical Conclusion Validity

Internal ValidityInternal Validity

External ValidityExternal Validity

Construct ValidityConstruct Validity

Context ValidityContext Validity



Interrupted TimeInterrupted Time--SeriesSeries

Sequence number

987654321

16

14

12

10

8

6

4

2

0

Probable and

Durable

Probable and

Non-Durable

Improbable

Regression to Mean

Improbable Constant

Change

Improbale Non-Linear

Change

Policy Intervention



Some Major Threats to ValiditySome Major Threats to Validity

HistoryHistory

MaturationMaturation

InstabilityInstability InstrumentationInstrumentation

TestingTesting

MortalityMortality

SelectionSelection

Regression towardRegression toward

the meanthe mean

ViolatedViolated

assumptionsassumptions

of statistical testsof statistical tests

Module 8 Monitoring Observed Outcomes

Documents

Transcript of Module 8 Monitoring Observed Outcomes