Roadway highway freeway exit signs billboards signs exit labels powerpoint presentation slides.
Exit presentation Program Logic An introduction Next Exit presentation.
-
Upload
montana-allcock -
Category
Documents
-
view
216 -
download
2
Transcript of Exit presentation Program Logic An introduction Next Exit presentation.
Exit presentation
Program LogicAn introduction
Next
Exit presentation
Exit presentation
This guide to program logic is designed for managers in government and community organisations, and for new evaluators.
Program managers can use program logic to plan and clarify how program activities lead to intended outcomes. Evaluators can use it to define key evaluation questions and shape methods for data collection. This guide covers:
• what is program logic• developing outcomes hierarchies • how to use program logic in evaluation
(c) 2009. This guide was first published in 1994 as a computer-based tutorial. We have refined it and release it again because people have told us that they still find it useful. The original drew on the work of the NSW Premier’s Department and in particular Sue Funnell in designing program logic for public sector programs in NSW. We also acknowledge the support of Jerome Winston.
Next
Exit presentation
Exit presentation
Menu
1. What is program logic?
2. Outcomes hierarchies
3. Using generic outcomes hierarchies
4. A program logic framework
5. Identifying evaluation questions
6. References
Back to introduction
Exit presentation
Exit presentation
1. What is program logic?
We need to start with some terminology.
“Program” refers to a set of coordinated activities and resources designed to achieve a desired result. A program can be large or small, simple or complex, and have a name such as “program”, “sub-program”, “project”, “initiative” and so on. However they all have the common feature of aiming to reach a goal, objective, result or outcome.
“Program logic” refers to the causal links between the elements of a program and its intended result or outcome. It shows the assumed chain of cause and effect between the activities and the desired result. By drawing out the logic as a diagram, we have a clearer picture of how the program is assumed to work.
Next Back to menu
Exit presentation
What is program logic?
In the evaluation literature, program logic is a form of program theory, which has been described as: • ‘a description of the mechanisms by which the program is expected to achieve its effects’ (Davidson, 2005)
• ‘an explanation of the causal links that tie program inputs to expected program outputs’ (Weiss, 1998)
• ‘mapping out a program’s inputs, activities, outputs, initial outcomes, intermediate outcomes, and longer term outcomes’ (Donaldson, 2007).
Sometimes it is called a logic model: • ‘... a way of portraying such theories concretely and visually’ (Frechtling, 2007)
• ‘...describe[s] a bounded project or initiative: both what is planned (the doing) and what results are expected (the getting). They provide a roadmap to a specific end’ (Knowlton and Phillips, 2009).
See references at the end of this guide.
Next Back Back to menu
Exit presentation
What is program logic?
Program logic is used to represent a program to help in planning or evaluation.
It is an analytical tool and not an end in itself. A program logic can be revised and refined to reflect changing focus on aspects of a program, or actual changes to the program over time.
For evaluation, drawing a program logic diagram can help indicate the key questions to ask about a program's performance at different times in the life of the program. It can be used to shape evaluative arguments about the success or otherwise of a program.
???????????????
√√√√√√√√√√√√√
Next Back Back to menu
Exit presentation
What is program logic?
Where does program logic fit in evaluation?
The evaluation of a program typically has four steps.
Identifying the program logic is usually part of the evaluation design (step 2).
It can also be used to support the conclusions in the reporting (step 4). 1.Plan the
evaluation
2.Design the
evaluation
3.Collect data
4.Analyse data and write a report
Next Back Back to menu
Exit presentation
What is program logic?
Analysing the program logic can help answer these questions : •how is the program meant to work – is it feasible?• to what extent is the program able to be evaluated at this time?• is the program strategy logical and appropriate?•what are key questions for the evaluation?•what information on program performance should be collected?
1.Plan the evaluation
2.Design the
evaluation
3.Collect data
4.Analyse data and write a report
Next Back Back to menu
Exit presentation
What is program logic?
As a starting point for developing a program logic, it can be useful to think about a program as a simple system with the stages illustrated below.
Environment
Inputs Processes Immediate results Long-term
results
Next Back Back to menu
Exit presentation
What is program logic?
Throughout this guide we will use an office skills program as a case study.
This can be represented by the diagram below.
Environment
Funds
Suitable courses
Young people
Community groups funded
Courses are run
Young people attain skills
Young people get jobs
Unemployment in the region is
reduced
Inputs Processes Immediate results
Long-term results
Next Back Back to menu
Exit presentation
What is program logic?
There are many reasons why a program may not be producing the intended results, for example
•it is being implemented incorrectly or •it does not meet the needs of the target group•the measurements being used are not able to show the results•the program logic is wrong
Environment
Funds
Suitable courses
Young people
Community groups funded
Courses are run
Young people attain skills
Young people get jobs
Unemployment in the region is
reduced
Inputs Processes Immediate results
Long-term results
Next Back Back to menu
Exit presentation
What is program logic?
Here we focus on the last reason - the program logic is wrong. In other words, some of the assumed links represented by the arrows are wrong.
In this case, attaining skills does not lead to young people getting jobs.
Environment
Funds
Suitable courses
Young people
Community groups funded
Courses are run
Young people attain skills
Young people get jobs
Unemployment in the region is
reduced
Inputs Processes Immediate results
Long-term results
XX
“Even if these young people have the skills, I wouldn’t give them a job unless their attitude was right.”
Employer
Next Back Back to menu
Exit presentation
What is program logic?
Program logic can be visually presented in different ways.
One useful way is an outcomes hierarchy.
It shows the “outcomes” a program is intended to produce at different levels, and the assumed links between them. The causal chain flows from the bottom outcome to the top. Outcomes hierarchies are explained in the next section.
Appropriate information gathered/
developed
Information organised into accessible form
Clients exposed to information
Increased awareness
Changed knowledge
Changed attitudes
Changed behaviour
Program specific outcomes
Next Back Back to menu
Exit presentation
What is program logic?
Program logic can also be presented in different ways to reflect different focus. Stakeholders may have their own ideas about how a program should work and describe the program in different ways.
Minister for Community Services
Course coordinatorProgram Manager – Department
“If we put more money into it we should get better results.”
“If these kids just turned up and did some work, they would have no trouble getting a job.”
“If the courses were run more efficiently, then more kids would be trained and get jobs.”
Next Back Back to menu
Exit presentation
What is program logic?
In practice, stakeholders have their own interests which can influence the way they think about the program’s logic. Working with stakeholders in designing and refining the program’s logic can help to build a better understanding of the program’s intended goals and the strategies needed to reach them.
Minister for Community Services
Program Manager
Next section Back Back to menu
Exit presentation
2. Developing outcomes hierarchies
A program can be described in terms of its intended outcomes. These outcomes can form a logical hierarchy, which can be used as a framework for evaluating the program.
Next Back to menu
Exit presentation
Developing outcomes hierarchies
Firstly consider a program’s intended results or outcomes, rather than its objectives or processes.
For example, for the office skills program (used as a case study in this guide):
An objective
To teach participants office skills
can be written as an outcome
Graduates have suitable office skills
An outcome is not a process... but a result that can be measured and observed
How many graduates have which skills?
Next Back Back to menu
Exit presentation
Developing outcomes hierarchies
While the office skills program was originally described by the Department’s manager in terms of aims, objectives and strategies, you can see that it aims to achieve a range of outcomes at different levels.
Unemployment is reduced
Aim:
Objectives:
Strategy:
To reduce long-term unemployment for target
group
To teach participants appropriate office
procedures
To assist course graduates to gain office
work positions
Design suitable course
Fund providers
Select students
Implement course
Graduates have suitable knowledge
and skills
A suitable office skills course is prepared
Participants react favourably to course
Next Back Back to menu
Exit presentation
Developing outcomes hierarchies
In fact, the office skills program aims to produce a whole series of outcomes. And we can see the outcomes will have logical links with each other based on cause and effect.
Graduates have suitable knowledge
and skills
Participants react favourably to course
For example,we have assumed this outcome has to be achieved first...
...in order to achieve the higher outcome.
Next Back Back to menu
Exit presentation
Developing outcomes hierarchies
The outcomes for the office skills program can be drawn up and arranged into a logical chain of cause and effect, like in this diagram.
The cause and effects chain is called an outcomes hierarchy. The logic is that each lower outcome has to be achieved in order to achieve the next higher outcome.
Long-term unemployment is reduced
Graduates have suitable knowledge and skills
Participants react favourably to course
Appropriate people apply for the course
Graduates gain office positions
Office skills course for target group is prepared
Information about training needs of target group is available
Next Back Back to menu
Exit presentation
Developing outcomes hierarchies
The outcomes hierarchy forms the basis of a program logic framework. Vertically, it shows the sequence of outcomes. Horizontally, it can be used to work out the details of outcomes. We will look at defining attributes, factors and performance information later in the guide (see section 4).
Outcomes Attributes of success
Factors Performance indicators
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Next section Back Back to menu
Exit presentation3. Generic outcomes hierarchies
Starting from a blank page to draw up an outcomes hierarchy for a program is not an easy task. Instead, a generic outcomes hierarchy can be a useful starting point.
A generic outcomes hierarchy can represent a whole range of different programs that have a common underlying logic or strategy.
This section describes some common generic outcomes hierarchies and how they can be used to develop an outcomes hierarchy for a specific program.
Next Back to menu
Exit presentation
Generic outcomes hierarchies
The Office Skills Program (the case study in this guide) can be classed as an educative program:
• it aims to influence people’s behavior to achieve its overall results
• it uses strategies involving communication and learning to change a target group’s knowledge, skills or attitude.
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Next Back Back to menu
Exit presentation
Generic outcomes hierarchies
The Quit Smoking Program represented by the outcomes hierarchy on the right is a very different educative program. It is a public education campaign conducted through the media. However, like the office skills program, it aims to influence people’s behaviour using educational strategies.
Outcomes
6. Decrease in smoking related disease
5. People quit smoking
4. Changed attitude to smoking in target group
3. Target group is exposed to the campaign
2. Media advertising
1. Information on smokers’ responses available
Next Back Back to menu
Exit presentation
Generic outcomes hierarchies
Although the outcomes for each program are very different, because they use the same kind of strategy, they can be represented by the same generic outcomes at each level.
Outcomes
6. Decrease in smoking related disease
5. People quit smoking
4. Changed attitude to smoking in target group
3. Target group is exposed to the campaign
2. Media advertising
1. Information on smokers’ responses available
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Program outcomes
Changed attitudes, skills or knowledge
Target group is exposed to the
information
Information is organised in
accessible form
Changed behaviour
Appropriate information is
available
Next Back Back to menu
Exit presentation
Generic outcomes hierarchies
These generic outcomes at each level can be used to form a generic outcomes hierarchy, which broadly represents all educative programs.
Program outcomes
Changed attitudes, skills or knowledge
Target group is exposed to the
information
Information is organised in
accessible form
Changed behaviour
Appropriate information is
available
Program outcome
Application of new skills, attitudes and knowledge
New skills, attitudes and knowledge
Program is implemented
Workable program is designed
Worthwhile goals are established
Next Back Back to menu
Exit presentation
Generic outcomes hierarchies
Generic outcomes hierarchies have been developed for seven types of programs, based on their overall strategy. These are:
• Educative• Regulatory• Case management• Advisory• Provision of a service/ product• Policy development• Revenue raising
These seven cover most public sector programs. Select from the list above to view a specific generic hierarchy.
Click here to skip the generic outcome hierarchies
Next Back Back to menu
Exit presentation
Generic outcomes hierarchies
EducativeAn educative program aims to influence people’s behaviour to achieve its overall objective.
It uses strategies which expose the target group to information and educational processes.
The exposure is intended to produce changes in the target group’s knowledge, skills or attitude, and, as a result, to change their behaviour.
Appropriate information gathered/ developed
Information organised into accessible form
Clients exposed to information
Increased awareness
Changed behaviour
Program specific outcomes
Changed knowledge, skills and/ or attitudes
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
RegulatoryA regulatory program aims to influence people’s behaviour to achieve its overall objective.
It uses strategies based on deterrence which require a regulation prohibiting unwanted behaviour.
The assumption behind the program is that potential offenders will be deterred if the target group is aware of both the prohibition and the risk of being caught for violating it.
Legislation
Licensing/ registration
Appropriate inspection and investigation
Identification of infringements
Appropriate behaviour, compliance
Program specific outcomes
Potential offenders are aware of the
program
Conciliation
Prosecution of people who fail
to comply
Potential offenders are
deterred
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
Case managementA case management program aims to influence people’s behaviour to achieve its overall objective.
It treats each case (person or group) individually, and sets objectives for them on a case-by-case basis.
The assumption behind a case management program is that as each individual achieves their own objectives, the circumstance of all the target group will improve and the overall program objective will be achieved.
Reduced long-term dependence and cost to government
Life circumstances improved
Short-term objectives progressively achieved
Selected individual programs satisfactorily implemented
Realistic objectives set on a case-by-case basis
Accurate identification of needs and capacity on a
case-by-case basis
Target group access program
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
AdvisoryAn advisory program aims to influence people’s behaviour to achieve its overall objective.
It uses strategies that encourage individuals in the target group to seek advice about an issue or concern.
The advice is intended to change their knowledge, skills or attitudes, and as a result, to change their behaviour.
Advisory programs differ from educative programs in that the target groups seek out the advice, rather than being exposed to it.
Appropriate information available
Information organised into accessible form
Target group aware of advisory service
Target group seek advice
Changed behaviour
Program specific outcomes
Changed knowledge
Changed attitudes
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
Service delivery A service delivery program aims to achieve its overall objectives through providing clients with a service or product.
A service provision program may be a non-competitive monopoly where the government is the sole provider, or the government may be in competition with other providers. The government may also fund NGOs to provide the service.
The clients of the program may vary from customers in the marketplace to clients with specific entitlements. The program outcomes could range from raising revenue to improved health for a target group.
Product/ service designed to match clients’ needs
Clients know about product/ service
Clients satisfied
Program specific outcomes
Clients use product/ service
Product/ service delivered efficiently
and effectively
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
Policy developmentA policy development program aims to change government policies to achieve its objectives.
It uses consultation and research to produce policy proposals which are realistic and acceptable to government and major stakeholders.
Information about issue is available
All stakeholders identified
Policies are approved by government
Desired change occurs
Process perceived as credible by government
Stakeholders support policy
proposals
Stakeholders’ views taken into account
Proposed policies are workable
Process perceived as credible by stakeholders
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
Revenue raisingA revenue raising program usually involves a legislative mandate, with strategies to make the target group aware of its responsibilities, and to collect the revenue.
Increased revenue
Monies collected
Licensing/registration
Clients aware of rights/ responsibilities
Legislation
Next Back to list of generic hierarchies Back to Menu
Exit presentation
Generic outcomes hierarchies
Once the initial hierarchy is designed, it can be improved by talking with stakeholders.
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
For example, in the case study office skills program, one stakeholder said: “Well these kids might pick up the skills, but half of them don’t want to work anyway.” And the outcomes hierarchy was refined (see diagram on the right).
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Graduates attain office skills
Graduates have suitable attitudes
Next Back Back to menu
Exit presentation
Outcomes
7. Long-term unemployment is reduced
6. Graduates develop their careers
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Graduates attain office skills
Graduates have suitable attitudes
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Graduates attain office skills
Graduates have suitable attitudes
Some of the participants in the office skills program said: “Now I’ve done the course, I never want to work in an office.” And the outcomes hierarchy was refined (see diagram on the right).
Refining the outcomes hierarchy helps build up a more realistic model of the program.
Generic outcomes hierarchies
Next Back Back to menu
Exit presentation
Generic outcomes hierarchies
A generic hierarchy can be a useful starting point for developing your outcomes hierarchy and for identifying the basic assumptions behind your program. Once you have talked to program staff and reviewed original program documents, you can se a generic hierarchy as a reference to help you begin drawing up your hierarchy.
In practice, many programs are more complex than a generic outcomes hierarchy.
You might find your program can be described by using two or more of the generic outcomes hierarchies.
The next section shows how to use an outcomes hierarchy to build a program logic framework
Next section Back Back to menu
Exit presentation
4. Program logic framework
Outcomes hierarchies can be used to build evaluation frameworks by looking at the details of the outcomes at each level.
Outcomes Attributes of success
Factors Performance indicators
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Next Back to menu
Exit presentation
A program logic framework
For example, the details of the Level 5 outcome for the Office Skills Program could be:
Outcomes Attributes of success
Factors Performance indicators
Level 5.Graduates have suitable knowledge, skills and attitudes
All graduates demonstrate satisfactory competencies
All feel confident to apply for office jobs
Cost per graduate is less than $700
Participants’ literacy levels
Teaching standards vary
Patterns of graduate test scores (gender, background)
Cost per graduate
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Attributes of success
Level 5.Graduates have suitable knowledge, skills and attitudes
Attributes of successWe will look at all the elements of the evaluation framework, starting with attributes of success. In the outcomes hierarchy, each outcome has only been described by a brief statement. It is helpful to identify specific characteristics of a successful outcome. These characteristics are called attributes of success for each outcome. They describe what the outcome is expected to look like if the program is operating successfully.
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Attributes of success
Level 5.Graduates have suitable knowledge, skills and attitudes
Attributes are defined by the needs of different stakeholders.
EmployerProgram Manager - Department
“Our pilots showed that 60% of this target group should learn the skills.”
“I’ll only employ people who know how to do the job properly.”
60% of graduates achieve satisfactory competencies
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Attributes of success
Level 5.Graduates have suitable knowledge, skills and attitudes
Key stakeholders will look for different attributes of success.
Students attending courses
Community administrator
“Our budget assumes that cost per graduate will be less than $700.”
“We want to feel confident when we go for job interviews.”
60% of graduates achieve satisfactory competencies
Cost per graduate is less than $700
Graduates build up their confidence in job seeking
Next Back Back to menu
Exit presentation
A program logic framework
FactorsFactors are the next element in the framework – factors affect how well the outcome is achieved. One way to consider the factors affecting the program is to do a SWOT analysis.
Strengths
Weaknesses
Opportunities
Threats
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Factors
Level 5.Graduates have suitable knowledge, skills and attitudes
Stakeholders can identify factors. For example, in the office skills program, teachers identify literacy as a weakness.
“You can’t expect all the kids to get the same office skills when half of them can’t read or write properly.”
Literacy levels of students
Course coordinator
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Factors
Level 5.Graduates have suitable knowledge, skills and attitudes
Different stakeholders are likely to suggest different factors. For example, the participants of the office skills program identify the teachers as a factor.
Students attending courses
“The teachers in the course were no good.”
Literacy levels of students
Suitability of teachers
Next Back Back to menu
Exit presentation
A program logic framework
Factors
Level 5.Graduates have suitable knowledge, skills and attitudes
It is important to be aware of factors when it comes to evaluating programs. It enables realistic targets to be set and provides a better understanding of the program’s results. Also, once important factors are identified, program managers can plan strategies to deal with them. In this way, they can improve the program and change their assumptions about the program’s logic.
Literacy levels of students
Suitability of teachers
Outcomes
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes
Level 5.Graduates have suitable knowledge, skills and attitudes
In the case study office skills program, some changes might be made based on factors.
“We want to know which teachers are producing the best results. In the future we might have to select our teachers more carefully.”
Community administrator
Factors
Literacy levels of students
Suitability of teachers
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes
Level 5.Graduates have suitable knowledge, skills and attitudes
Another change might relate to the literacy level of students.
Factors
Literacy levels of students
Suitability of teachers
“To account for individual differences, we record each students’ progress separately. We are considering introducing a basic literacy course as well.”
Teacher
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Factors
Level 7. Long-term unemployment is reduced
For higher level outcomes, there may be many more factors outside the control of the program. Direct links between these outcomes and the program may be difficult to see.
Rapid technological changes
Economic restructuring
Increased migration
Next Back Back to menu
Exit presentation
A program logic framework
Performance informationThe attributes of success and factors identified in the last two steps give a more detailed picture of each outcome. These can be used to work out what information is needed to see whether the outcome is being achieved – the performance information. The attributes of success and factors identified in the case study suggest a need for information on:
• how well graduates learnt about office work• the quality of teachers in the program• the cost effectiveness of the course.
Outcomes Attributes of success
Factors Performance indicators
Level 5.Graduates have suitable knowledge, skills and attitudes
60% of graduates achieve satisfactory competencies
Graduates build up their confidence in job seeking
Cost per graduate is less than $700
Participants’ literacy levels
Teaching standards vary
Patterns of graduate test scores (gender, background)
Profile of teachers
Cost per graduate
Next Back Back to menu
Exit presentation
A program logic framework
Outcomes Factors
Level 6. Course graduates get jobs
The program logic framework indicates what performance information is needed to assess how well the program is achieving a specific outcome. This information can come from program monitoring or an evaluation study. However, data collection can raise issues. For example, collecting certain data could be costly, unethical, or ultimately not very worthwhile because of the impact of other factors.
“That information would be great to know, but who has time to chase up the kids?”
Patterns of graduate employment after the course
Data: % of graduates who remain in the job for 3 months (by type)Course coordinator
Next Back Back to menu
Exit presentation
5. Identifying key questions
In the previous section we built a program logic framework based on an outcomes hierarchy. Collecting information on all outcomes could be expensive and inefficient, and priorities need to be set. When we design an evaluation or monitoring system, the framework can be used to decide which outcomes to focus on, what questions to ask, and what information is needed. The issues to consider in deciding on the key evaluation questions to ask include: the purpose of the evaluation the balance between efficiency, effectiveness, appropriateness and equity the interests of stakeholders any issues that have arisen from monitoring the program.
Next Back to menu
Exit presentation
Identifying key questions
While evaluations are used for many different purposes, most of them are a balance between:
Program improvement
Program justificationand
What changes are needed? Should the program continue?
Next Back Back to menu
Exit presentation
Identifying key questions
Evaluations may include questions that focus on:
Efficiency
Effectiveness
Appropriateness
Equity
Adequacy
Outputs achieved related to inputs.
The extent that outcomes match what was intended.
How well the program aimed to meetthe problem it set out to address;the needs of the client group;and how well it fit government priorities.
How fair the program was to all people in the target group.
Whether the response was sufficient given the problem or need.
Next Back Back to menu
Exit presentation
Identifying key questions
Stakeholders might see different purposes for an evaluation. For example, in the office skills program, the program manager wants the evaluation to answer the following questions.
Program Manager - Department
“I want to know how well the program is being run, and how we can do better next year.”
“Are my budget targets being met?”
“Are all the kids getting a fair chance?”
Program improvement
Efficiency
Equity
Next Back Back to menu
Exit presentation
Identifying key questions
While the course coordinator wants to know about:
Effectiveness
“I wonder if the course structure is right. Would it be improved by more work experience time?
Course coordinator
Program improvement
Next Back Back to menu
Exit presentation
Identifying key questions
The Minister for Communities wants to know about:
Effectiveness
“Most of the cabinet thinks that these programs are a waste of money. Some say that government should not be involved at all.
I want to know whether these programs actually have any impact on unemployment.
And I want some good success stories to show the media.”
Minister for Community Services Appropriateness
Program justification
Next Back Back to menu
Exit presentation
Identifying key questions
The educational consultant wants to know about:
Effectiveness
“Classroom-based programs can’t give people real life skills, and using an educational model just won’t work with that target group.”
Educational consultant Appropriateness
Program justification
Next Back Back to menu
Exit presentation
Identifying key questions
The purpose of the evaluation will direct you to relevant questions. At the same time, different stakeholders will have their own interests.
“What is our cost for each person who gets a job?”
Program Manager - Department
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
“Are the right people applying?”
Next Back Back to menu
Exit presentation
Identifying key questions
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
“Does our course cover all of the competencies you need for office work?”
Course coordinator
Next Back Back to menu
Exit presentation
Identifying key questions
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
“Will these kids have the right skills and attitudes?”
Employer
Next Back Back to menu
Exit presentation
Identifying key questions
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
“What are the changes in youth unemployment levels?”
Minister for Community Services
“How many got jobs?”
Next Back Back to menu
Exit presentation
Identifying key questions
Stakeholders’ interests may vary, and be at different levels of the hierarchy.
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
Politicians and senior management are usually interested in the higher level outcomes where they are seen as accountable.
Minister for Community Services
Program staff may want to focus on lower level outcomes where they are familiar and have more control over the results.
Course coordinator
Next Back Back to menu
Exit presentation
Identifying key questions
Outcomes
7. Long-term unemployment is reduced
6. Graduates get office jobs
5. Graduates attain office skills
4. Participants react favourably to the course
3. Appropriate people apply for course
2. Office skills course for target group is prepared
1. Information on training needs of target group is available
When you are planning an evaluation, the
framework makes it easier to consider a range of stakeholder
viewpoints
Next Back Back to menu
Exit presentation
Identifying key questions
Outcomes Performance information
Level 6. Course graduates get jobs
Regular program monitoring can also raise further questions for an evaluation.
“Our records show that 80% of the graduates who get jobs are men. Yet, equal numbers of men and women enrol. What’s going on?”
% of graduates who remain in the job for 3 months
Program Manager - Department
End Back Back to menu
Exit presentation
You have reached the end of this tutorial – you can return to the main menu to review of the sections, continue to a list of references, or exit the presentation.
References Back to menu
Exit presentation
References
Frechtling, J.A, 2007, Logic Modeling Methods in Program Evaluation, Wiley, Inc., Jossey-Bass, San Francisco Torvatn, H., 2008, Book review (Frechtling, Logic Modeling Methods in Program Evaluation), Evaluation and Program Planning
W.K. Kellogg Foundation, 2001, Logic Model Development Guide, Using Logic Models to Bring Together Planning, Evaluation & Action
Wyatt Knowlton, L and Phillips, C 2008, The Logic Model Guidebook, SAGE
Rogers, P.J., Hacsi, T.A., Petrosino, A. & Huebner, T.A (editor), 2000. Program Theory in Evaluation Challenges and Opportunities: New Directions for Evaluation, No. 87, Jossey-Bass.
Weiss, Carol H. 1998, Evaluation: Methods for Studying Programs & Policies - 2nd edition, Prentice Hall
Davidson, D.E.J., 2004. Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation 1st ed., Sage Publications, Inc.
Back to menu