Download - Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Transcript
Page 1: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Policy Analysis and

Decision Makingwith Emphasis on Chronic Non-

communicable Diseases

Cristina Puentes-Markides

LESSON 4 : INSTRUMENTS OF PUB L IC ACCOUNTAB IL ITY AND POL ICY CHANGE

Page 2: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

2

CPM-2012

Public accountability (Functions of monitoring and evaluation)

Policy change Agenda setting Advocacy Policy learning

Key Points

Page 3: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Generally, public health policies are evaluated through health programs and plans.

National/local health plans need to include the corresponding programs (with the appropriate goals, objectives, measures and targets) that address the priority NCDs.

The products of evaluation are useful only when they inform and guide decision making, and when they complement monitoring activities.

Monitoring and evaluation activities within established performance frameworks of production and reporting contribute to public accountability We can all participate in modifying policies through various means.

Key messages3

CPM-2012

Page 4: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Accountability is defined as “an obligation or willingness to accept responsibility or to account for one's actions.” (Merriam Webster Dictionary)

Therefore, being accountable means that one is required to explain actions or decisions to others.

WHAT IS ACCOUNTABILITY?

4

CPM-2012

Page 5: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

MONITORING AND EVALUATION

Page 6: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Monitoring and evaluation are no longer simple measurement exercises.

Increasing requirements for modern societies: transparency and accountability of governments, demands by better informed and organized citizens.

Media and communication/information technologies have contributed to increase the public’s expectations, channel their opinions and demands.

INCREASING IMPORTANCE OF M & E

6

CPM-2012

Page 7: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Timely and reliable monitoring and evaluation can contribute to the identification of areas of improvement and thus to better quality and results.

Reporting that informs decision-making and management activities improves performance.

Accountability and compliance are also enhanced (demonstrates whether the duties and responsibilities have been conducted according to standards & procedures established by legislation, regulatory agencies and/or professional associations).

M & E can explain why the results and outcomes of a particular policy or program are what they are, why they differ between groups or localities.

BENEFITS OF MONITORING AND EVALUATION

7 Is the policy change effected having the impact it was meant to have?Have there been any unintended consequences of the policy which detract from its overall effectiveness?Could the policy be improved further?Were the costs (financial, political, etc.) spent on the change worth the resulting benefits?What lessons were learned in effecting the change that could be used to improve the effectiveness of future policy advocacy efforts?

Page 8: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Process that registers changes in the key variables after implementation of a particular policy or program; documents whether there were changes due to the policy or program.

Requires the identification of variables, targets and indicators to measure changes, and the means to assess whether the process is free of bias (e.g. related to the design, the process, influence of those who support or those that are against that particular policy/program, etc.)

MONITORING8

CPM-2012

Page 9: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

“Evaluation is concerned with establishing the value premises necessary to produce information about the performance of policies.”

  “… evaluation refers to the production of information about the value or worth of policy outcomes.” (Dunn, W. 1994. Public Policy Analysis. 2nd Ed).

The purpose of evaluation is to measure and explain the effects of a program against the goals it set out to accomplish, providing information for later decisions about the program

EVALUATION9

Program evaluation is a subset of policy studies, and a such, of great importance.

Page 10: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

10

CPM-2012

Recognize the moral debate and that everyone has the right to their legitimate opinions.

Recognize that there is power, lack of power and empowerment.

Develop a genuine dialogue between organizations and users.

Promote openness to criticism, complaints, questions.

Enable and promote that groups give their opinion about the program/policy.

GOOD PRACTICES OF EVALUATION (Everitt, 1996)

Democratic and honest processes of evaluation are essential to ensure good and fair practices.

Page 11: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

11 Evaluation, as the application of social

sciences to human problems is paved with decisions. (Weiss).

Policy denotes relationships of power and influence and some type of resource allocation by those individuals/entities that have authority over those resources.

Some stakeholders and relevant actors have particular interests about what type of evaluative data/information they are interested in and want to be public.

YET, EVALUATION IS ALSO POLITICAL

See Weiss, C. H. Where politics and evaluation research meet. In: D. J. Palumbo. Ed. The politics of program evaluation. Newbury Park: Sage. 1987.

Page 12: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

12

BENEFITS AND POSSIBLE CONSTRAINTS OF EVALUATION

BENEFITS CONSTRAINTS Informs about policy

performance, and the goods and services received by he beneficiaries.

Inadequate levels of human and financial resources assigned to evaluation activities.

Contributes to clarify underlying values in the selection of alternatives, goals and objectives.

Personnel with evaluation skills may not be available.

Results can be useful inputs for future policy analyses or for the design of new programs.

The objectives and targets for certain programs are difficult to measure.

The actual change in behavior and obtained results reflect the policy impact.

Measuring the impact of certain interventions may result to be challenging and costly.

The use of evaluation results in decision-making is limited.

“Evaluation is integrated into every phase of the health policy development cycle to guide decision making and improve effectiveness. Evaluation is of limited use if it only provides feedback after the fact. For optimal use, evaluation is therefore integrated into the quality management processes which support the health policy development cycle.” Assessing the Effects of Public Health Policies and Actions. Evaluation Strategic Framework. Federal Office of Public Health, Switzerland. December 2005.

Page 13: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

13

Major Types of EvaluationQuestions that the evaluation type

seeks to answerFormative Evaluation: Aims at continuous program improvement; collects different types of information about program activities outputs client satisfaction obstacles to implementation. May include needs assessments, evaluability assessments, implementation evaluation & process evaluations.

Asks what aspects of our situation most shaped our ability to do the work we set out to do in our community? What did out program accomplish in our community.

Summative Evaluation: Seeks to learn whether the policy program or intervention actually works focus on demonstrated program outcomes and impacts only. Monitoring is a concurrent activity. Sometimes they are also called "impact" evaluations but can also include outcome evaluations, CEA & CBA secondary and meta-analysis.

What is out assessment of what resulted from our work in the community? What have we learned about doing this kind of work in a community like ours?

Impact Evaluation: A form of outcome evaluation. It focuses on the program outcomes and impacts, assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. IE is employed when external factors are known to influence the program’s outcomes in order to isolate the program’s contribution to achievement of its objectives.

Seeks to answer whether the program worked or did it have its intended effects? If so who was helped and what activities or characteristics of the program created the impact? Did the program have any unintended consequences positive or negative?

Process Evaluation: Assesses the extent to which a program is operating as it was intended. It assesses program activities’ conformance to statutory and regulatory requirements, program design and professional standards or client/customer expectations. Documents the procedures and activities undertaken in service delivery, responds to questions about how the program operates, helps identify problems faced in service delivery. Useful to practitioners and service providers in replicating or adapting program strategies.

How well is the program being implemented and what are the barriers to implementation? How do beneficiaries/clients use program services? What does the program do well? What is not being done well? What are typical complaints from staff and/or students/clients? Are established program policies and procedures being followed? Are program resources being used efficiently?

Cost Evaluations: Address how much the program or program components cost preferably in relation to alternative uses of the same resources & to the benefits being produced by the program. When applied to existing programs, CBA and CEA are considered forms of program evaluation.

CEA assesses the cost of meeting a single goal or objective and can be used to identify the least costly alternative to meet that objective effectively. CBA aims to identify all relevant costs and benefits related to the program usually expressed in dollar terms.

Page 14: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

NEED TO BE FAMILIAR WITH CERTAIN CONCEPTS RELATED TO PERFORMANCE AND TO EVALUATION

14

CPM-2012

Page 15: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Develop a logic model to establish causal linkages between the policy/programs and the goals/outcomes. (see further)

DEVELOP A LOGIC MODEL15

CPM-2012

Page 16: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

16

is a graphic instrument that represents a “logic" chain of the theory and actions of the program to be designed & evaluated; tests whether the policy, program or initiative makes sense.

Clarifies linkages between inputs to linking inputs to results and outcomes expected form a policy, program or initiative.

Represents the framework that sustains performance measurement & evaluation strategies.

Useful to summarize & communicate about complex programs, their rationale, activities and expected results of the policy or program.

A LOGIC MODEL

Inputs Process Outputs Outcomes

Page 17: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Can standardize activities and outcomes

Great guide for evaluation (just develop data collection around outputs and outcomes)

Essential for program planning and evaluation.

Identifies moments of the evaluation, gaps and uncertain assumptions.

LOGIC MODELS ARE TIME CONSUMING, BUT

17

CPM-2012

Page 18: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

● What is measured can be improved.

● This is true only if we are measuring the right things.

FUNDAMENTAL PRINCIPLES 18

Page 19: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Be clear about your measuresTypes of Measures

INPUTS• Resources needed to provide a particular product or service, may represent demand factors (e.g. characteristics of target populations), conditions under which care is provided (material (facilities, equipment); human (ratios, qualifications, experience); organizational characteristics (size, volume, IT system)

PROCESS

•What is done to, for, with, or by defined individuals or groups as part of the delivery of services (screening and diagnosis, treatment and rehabilitation, education and prevention)

OUTPUTS•Products, services and events intended to lead to outcomes, linked to problems or issues to be addressed, and associated with an activity. Do not indicate whether goals have been accomplished, or reveal quality or efficiency of service provided.

OUTCOMES

• Measurable results or changes attributable to a specific policy or set of policies. Changes (or lack thereof) in the health of a defined population related to an intervention (health status – morbidity/ mortality, complications, readmissions, social functioning, consumer satisfaction), changes attributable to health care. Reflect whether the service meets its goals and whether actual results are achieved.

Page 20: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

E. Taylor Powell, Univ. of Wisconsin Extension. 200320

INPUTS   OUTPUTS   OUTCOMES - IMPACT

 Activit

ies Particip

ation  Short term

Medium Term

Long term

  

What we invest

 What

we do Who we reach

 What the

short term results are

What the medium

term results

are

What the ultimate impact(s)

are

   

ASSUMPTIONS   EXTERNAL FACTORS     

EVALUATIONFocus – Collect Data-Analyze and Interpret - Report

Situatio

n

Priorities

Planning: Start with the end in mind

Page 21: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

E. Taylor Powell, Univ. of Wisconsin Extension. 200321

INPUTS   OUTPUTS   OUTCOMES - IMPACT

  Activities Participa

tion  Short term

Medium Term

Long term

                 

What we invest

 What we do Who we reach

 

What the short term

results are

What the

medium term

results are

What the ultimate impact(s

) are

StaffVolunteersTimeMoneyResearch baseMaterialsEquipmentTechnologyPartners

 

Conduct workshopsDeliver servicesDevelop products, curriculum, resourcesTrainProvide counselingAssessFacilitatePartnerWork with needs 

ParticipantsClientsAgenciesDecision makersCustomersSatisfaction

 

LearningAwarenessKnowledgeAttitudesSkillsOpinionsAspirationsMotivations

ActionBehaviorPracticeDecision-makingPoliciesSocial Action

ConditionsSocialEconomicCivicEnvironmental

ASSUMPTIONS   EXTERNAL FACTORS     

EVALUATIONFocus – Collect Data-Analyze and Interpret - Report

SituationNeeds and assets

Symptoms vs. problemsStakeholder engagement

Prioritiesmission, vision, values,

mandates, resources,

local dynamics,

collaborators, competitors

Intended outcomes

Planning: Start with the end in mind

Page 22: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

POLICY CHANGE

Page 23: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Policy is not the sole prerogative of institutional agents

We can all participate in the process to influence and change policy.

Population can shape policy through various means.

PARTICIPATION IN POLICY CHANGE

23

CPM-2012

Page 24: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Voting Specific work through various policy areas As elected representatives As policy managers in government As academic researchers As journalists or in other roles in the media In professional and/or community

organizations In political parties

MANNERS OF PARTICIPATION IN POLICY CHANGE

24

CPM-2012

Page 25: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Agenda setting Advocacy Coalition building Policy learning

Activities that support policy change

25

CPM-2012

Page 26: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

“Separate streams come together at critical times. A problem is recognized, a solution is developed and available in the policy community, a political change makes it the right time for policy change, and potential constraints are not severe. These policy windows, the opportunities for action on given initiatives, present themselves and stay open for only short periods”

According to Kingdon, J.

26

Kingdon, J. 1995.

•Policy change is likely when a problem is widely recognized, a solution is either swiftly developed or is already available, and the politics are favorable. •Policy change is unlikely if the problem goes unrecognized, or if a solution is lacking, or if the politics are unfavorable.

Page 27: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Kingdon’s Model of Agenda-setting

27

PR

OC

ESS

STR

EA

MS

PROBLEM STREAM

(Information & events that may unchain a series of

actions related to placing or eliminating an issue from the

agenda).

Indicators, focusing events, definitions,

values, comparisons, categories

POLICY STREAM(Knowledge & advice derived from researchers, advisors &

analysts that offer alternatives/solutions that

may or may not be considered or used by decision makers)

Alternatives, policy communities,

entrepreneurs, criteria for survival, technical feasibility, value

acceptability, political feasibility, emergence of

consensus

POLITICS STREAM(The will of the political

system & actors to place an issue on the agenda)

Power, resources, symbolism, timing,

national mood, organized forces.

Page 28: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Kingdon’s Model: Streams come together

28

PROBLEM STREAM

POLICY STREAM

POLITICS STREAM

Window (opportunity for

advocates of proposals to push their solutions or attention to their special issues.)

T i m e

Page 29: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Kingdon first coined the term and defines policy entrepreneur as “advocates for proposals or for the prominence of ideas” (1995, 122).

Analogy of a surfer, waiting on the board to catch the wave (ODI) Entrepreneurs can “be in or out of government, in elected or appointed positions, in interest groups or research organizations”.

Willing to “invest their resources - time, energy, reputation and sometimes money - in the hope of a future return”

Policy entrepreneurs play a key role in setting policy agendas; they manipulate an issue to achieve their desired policy change.

POLICY ENTREPRENEURS

29

CPM-2012

Page 30: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Source: ODI30

DIFFERENT TYPES OF POLICY ENTREPRENEURS

Storytellers

Engineers

Networkers

Fixers

Practitioners, bureaucrats and policy-makers often articulate and make sense of complex realities through simple stories. Sometimes misleading, yet their narratives are very powerful.

Policy-making usually takes place within communities of people who know each other and interact. If you want to influence policymakers, you need to join their networks.

Often, a huge gap between what politicians and policy-makers say they are doing and what actually happens. Researchers/ analysts need to work not just with the senior level policy-makers, but also with the 'street-level bureaucrats'.

Policy making is essentially a political process. You don’t need to be a Machiavelli, but successful policy entrepreneurs need to know how to operate in a political environment - when to make your pitch, to whom and how.

Page 31: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Integrate Participation in the Policy Cycle

Source: C. Sessa, Institute for the Integration of Systems, Rome. 31

•Policy evaluation

•Identification of problems and objectives

•Policy Implementation

• Policy formulation

Information provision: introduction of plans/ projects; incentives for participationInformation exchange: discussion of planning drafts

Information/provision: announcement, awareness raising.Deliberation: implementation of plans, solution of conflicts

Information provision: awareness raising.Information gathering: evaluation of the outcome.

Information provision: introduction of plans/ projects; incentives for participation.Information gathering: problem definitionInformation exchange: problem definition

Page 32: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

LEVEL OF STAKEHOLDER PARTICIPATION AT DIFFERENT STATES OF THE POLICY PROCESS

Adapted from FAO. Easypol. Module 175. Policy Processes : Part 3. Two Important Aspects of Government-Citizen Synergy in Policy Processes. Participation and Communication. 32

STAKEHOLDER GROUPS

AGENDA SETTING

POLICY FORMULATIO

N

IMPLEMENTATION

MONITORING &

EVALUATION

National Level        

Government        

Representative assembles

       

General public        

Poor & vulnerable groups

       

Organized civil society

       

Private sector        

Donors        

Local level        

Government        

Representative assembles

       

General public        

Poor & vulnerable groups

       

Organized civil society

       

Private sector        

Donors        

Page 33: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

ADVANTAGES OF PARTICIPATORY METHODS AT DIFFERENT STATES OF THE POLICY PROCESS

Adapted from FAO. Easypol. Module 175. Policy Processes : Part 3. Two Important Aspects of Government-Citizen Synergy in Policy Processes. Participation and Communication. Adapted from World Bank, 2004. 33

AGENDA SETTING FORMULATION

IMPLEMENTATION, MONITORING &

EVALUATION

Combining datasets and involving a range of stakeholders leads to a more realistic understanding of policy issues and more appropriate actions

Opens dialogue

Builds broad consensus

Maximize impact on the relevant beneficiaries/clients - participation gives a range of options that increase the possibility of success

Country ownership Transparency and accountability

Develops trust between government and civil society

Adapt institutional arrangements accordingly  Increases relevance

and probability of successful implementation

Page 34: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Different types/levels of participation and example of associated techniques

Source: FAO. Easypol. Module 175. Policy Processes : Part 3. Two Important Aspects of Government-Citizen Synergy in Policy Processes. Participation and Communication.

34

TYPES/LEVEL OF

PARTICIPATION

EXAMPLES OF PARTICIPATION TECHNIQUES

TYPE OF IMPACT

INTENSITY OF

PARTICIPATION

Information: one way flow

From government to public: public information, press conferences.From public to government: questionnaire, surveys, toll-free telephone hotlines.

Impact on stakeholders’ level of information and knowledge but no influence on who decides.

Low

Consultation: two way flow and exchange of views

Public hearings, government –led working groups, workshops, field trips

Shared decision-making: shared control over decisions

Joint committees, advisory councils, task forces.

Impact before or on decision

Shared decision and actions.

Negotiation, participatory budgeting, co-management of natural resources.

Impact on decision High

* The intensity of participation is linked to the type of impact participation has on decision-making

Page 35: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

The active support to a particular cause Attempts to gain support for that cause from

other individuals and/or groups. Calls attention to an issue that is deemed

important, and ensuing efforts to shift the attention of decision-makers to address this issue.

ADVOCACY 35

UNFPA. Guidelines. Support for Advocacy, 1997.

In public health, advocacy is the deliberate participation of public health practitioners in the process of policy

formulation to achieve better health results.

Page 36: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

As government bureaucrats : Covert public health advocacy: tobacco control, road safety; countering the advocacy of people who are opposed to public health policy: anti-immunization, anti-fluoridation.

As members of public health organizations: Planning, funding and steering of policy advocacy campaigns: tobacco control, screening, immunizations.

As staff in community health agencies: Support community networks organize and voice their concerns about their health, toxic waste dumps, hospital closures, lack of access to certain services.

As members of professional bodies : Organizing and participating in campaigns to achieve change, behavioral counseling, licensing of traditional medical practitioners, access to care.

As members of self-help and consumer/interest groups (Diabetes, epilepsy, Alzheimer's, cancer, special services or homes.)

As activists in social movements beyond health: Chronic illnesses, HIV/AIDS councils for expediting drug approval and distribution.

IN THE SPECIFIC CASE OF PUBLIC HEALTH YOU CAN ENGAGE

36

Legge, D. Policy Advocacy. La Trobe Univers. Australia

Page 37: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Prepare Research and specific policy proposals Communications material (newsletters, reports, press

releases) Evidentiary support (case studies, market analysis, impact

reports) Persuade decision-makers, opponents, the public Publicise

to frame the debate to build consensus indirectly to educate

Mobilise Beneficiaries, “Champions” Partners Supporters

Build consensus

This may mean37

Legge, D. Policy Advocacy. La Trobe Univers. Australia

Page 38: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

ADVOCACY AREAS AND INSTRUMENTS

CPM-2012 38

Leadership developmentCoalition-buildingEstablishment of networksCapacity-buildingMobilization of communitiesPolitical lobbyingPromotion of legislative changesInformation to the mass mediaCounter the opposition

Direct lobbying and link with decision-makers (legislators, cabinet members, other), interest groups and individuals with influence and power over the particular topic of interest.

Public relations Social marketing Mass media and

related links Direct action

(boycotts, demonstrations, etc.)

Page 39: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

POLICY LEARNING

“Policy learning is defined as all those processes by which

policy systems generate and incorporate knowledge and understanding about

i.underlying causes and conditions of policies and initiatives

ii.the effects of policy and initiatives.

This knowledge is derived throughout the policy cycle and policy learning provides feed-back to all stages“

(Hjelt et al., in OECD 2005: Governance of Innovation Systems, p.196)

Wolfgang Polt-Joanneum Research. Institute for Technology and Regional Policy. The role of policy advisers in policy learning. 6CP Conference. 23-24 May 2006

39

Page 40: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

BARRIERS TO POLICY LEARNING• Why look back at old programmes /

initiatives? We are forward looking !“• Evaluations are dangerous because of

potential negative outcomes• Not ‚making‘ an experience, is not learning at

all !

RELUCTANCE TO

CONFRONT THE PAST

• Communicating in different ‚codes‘• Communicating not at all• Not accepting or understanding different

perspectives

COMMUNICATION

FAILURE(S)

• Policy stripped of ressources• Information overflow• Coordination overload • Not being able to capitalize on available

information

LACK OF ADOPTION CAPACITY

• Loss of personel• Institutional shake-ups• Not being able to retain and pass on available

knowledge !

LOSS OF INSTITUTIONAL MEMORY

Wolfgang Polt-Joanneum Research. Institute for Technology and Regional Policy. The role of policy advisers in policy learning. 6CP Conference. 23-24 May 2006

Page 41: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Recognizing how the policy/political climate shapes effectiveness.

Recognizing frames, perceptions.

Awareness of complexity and potential tensions

Recognizing and taking advantage of policy windows.

A policy entrepreneur, someone willing to invest time and energy for policy change

Ability to integrate different types of knowledge for policy relevance.

Power, position

Reputation, prestige, political clout, credibility (ethos), ability to appeal to the emotions of the audience (pathos) and ability to present good reasons/logic (logos).

ULTIMATELY, A SUCCESSFUL POLICY ARGUMENT LEADING TO CHANGE DEPENDS PARTLY ON41

CPM/2012

Page 42: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

THANK YOU!!

42

Page 43: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

SOME USEFUL CONCEPTS RELATED TO PERFORMANCE

Source: Turning Point Collaborative

43

Performance: refers to results obtained from processes and administrative and technical services that allow monitoring, evaluation and comparison relative to goals, standards, past results and other organizations.

Performance Standard: Standards are one form of performance measure; they are generally objective standards or guidelines that are used to assess performance.

Performance Target: The planned or expected level of performance.

Performance Measure: Qualitative or quantitative features of performance used to measure results & ensure accountability; report on the achievement of the objectives & effectiveness of a program. May address the type or level of program activities conducted (process), the activities, products and services (outputs), and/or the benefits derived (outcomes). They are conceptual, and need to be operationalized, they have targets.

Performance Indicator: “provides a concrete measure of a specific capacity, process or outcome related to an accountable entity that is part of a defined health improvement strategy for a specific issue (IOM, 1997); they represent the data or information used to assess progress toward a performance standard; are operationalized, specify actual data (qualitative or quantitative) to be collected and used and do not have targets.

Page 44: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

Performance measurement is the ongoing, regular analysis of periodically collected data for monitoring how a policy, program or initiative is doing at any point in time. May be carried out by a program, an organization or an agency that reports on the level of attainment of pre-established goals (planned results) and any gaps between those planned and those achieved over time.

Performance monitoring (often compared to the dashboard of a car, which gives regular readings) helps assess the policy, program or plan performance by informing policy makers, service providers, managers and/or donors about the degree of achievement of specific objectives..

Program evaluation is more inclusive. It answers the "why" questions about a program, it informs about how well a program delivers and how the outcomes relate to the inputs or outputs of the program. Program evaluations may be prospective (e.g. policy analyses, evaluability studies, program or project assessments) or retrospective (e.g. health services research, process or outcome evaluations).

Performance measurement/monitoring and program evaluation: different but complementary44

CPM-2012

Page 45: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

45

Major Types of EvaluationQuestions that the evaluation type

seeks to answerFormative Evaluation: Aims at continuous program improvement; collects different types of information about program activities outputs client satisfaction obstacles to implementation. May include needs assessments, evaluability assessments, implementation evaluation & process evaluations.

Asks what aspects of our situation most shaped our ability to do the work we set out to do in our community? What did out program accomplish in our community.

Summative Evaluation: Seeks to learn whether the policy program or intervention actually works focus on demonstrated program outcomes and impacts only. Monitoring is a concurrent activity. Sometimes they are also called "impact" evaluations but can also include outcome evaluations, CEA & CBA secondary and meta-analysis.

What is out assessment of what resulted from our work in the community? What have we learned about doing this kind of work in a community like ours?

Impact Evaluation: A form of outcome evaluation. It focuses on the program outcomes and impacts, assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. IE is employed when external factors are known to influence the program’s outcomes in order to isolate the program’s contribution to achievement of its objectives.

Seeks to answer whether the program worked or did it have its intended effects? If so who was helped and what activities or characteristics of the program created the impact? Did the program have any unintended consequences positive or negative?

Process Evaluation: Assesses the extent to which a program is operating as it was intended. It assesses program activities’ conformance to statutory and regulatory requirements, program design and professional standards or client/customer expectations. Documents the procedures and activities undertaken in service delivery, responds to questions about how the program operates, helps identify problems faced in service delivery. Useful to practitioners and service providers in replicating or adapting program strategies.

How well is the program being implemented and what are the barriers to implementation? How do beneficiaries/clients use program services? What does the program do well? What is not being done well? What are typical complaints from staff and/or students/clients? Are established program policies and procedures being followed? Are program resources being used efficiently?

Cost Evaluations: Address how much the program or program components cost preferably in relation to alternative uses of the same resources & to the benefits being produced by the program. When applied to existing programs, CBA and CEA are considered forms of program evaluation.

CEA assesses the cost of meeting a single goal or objective and can be used to identify the least costly alternative to meet that objective effectively. CBA aims to identify all relevant costs and benefits related to the program usually expressed in dollar terms.

Page 46: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

LOGIC MODELS

Examples

Page 47: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

CPM-201247

Page 48: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

CDC Logic Model for integrating mental health into chronic disease prevention and health promotion.48

Page 49: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

E. Taylor-Powell. Univ. of Wisconsin, Extension. 200349

State level logic model: Reducing and preventing youth tobacco use

Page 50: Cristina Puentes-Markides L ESSON 4: I NSTRUMENTS OF P UBLIC A CCOUNTABILITY AND P OLICY C HANGE.

CPM-201250