Evaluation: What is it and why do it?

Evaluation. What associations does this word bring to mind? Do you see evaluation as an invaluable tool to improve your program? Or do you find it intimidating because you don't know much about it? Regardless of your perspective on evaluation, MEERA is here to help! The purpose of this introductory section is to provide you with some useful background information on evaluation.

Table of Contents What is evaluation?

Why should I evaluate my program?

What type of evaluation should I conduct and when?

What makes a good evaluation?

How do I make evaluation an integral part of my program?

How can I learn more?

What is evaluation?

Evaluation is a process that critically examines a program. It involves collecting and analyzing information about a programs activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions (Patton, 1987).

Why Should I evaluate my program?

Experts stress that evaluation can:

There are some situations where evaluation may not be a good idea!

when the program is unstable, unpredictable, and has no consistent routine

when those involved cannot agree about what the program is trying to achieve

when a funder or manager refuses to include important and central issues in the evaluationAdapted from Thomson and Hoffman (2003).

Improve program design and implementation.It is important to periodically assess and adapt your activities to ensure they are as effective as they can be. Evaluation can help you identify areas for improvement and ultimately help you realize your goals more efficiently. Additionally, when you share your results about what was more and less effective, you help advance environmental education.

Demonstrate program impact. Evaluation enables you to demonstrate your programs success or progress. The information you collect allows you to better communicate your program's impact to others, which is critical for public relations, staff morale, and attracting and retaining support from current and potential funders.

Why conduct evaluations?, approx. 2 minutes

Gus Medina, Project Manager, Environmental Education and Training Partnership

What type of evaluation should I conduct?

Make evaluation part of your program; dont tack it on at the end!

The evaluation experience is likely to be more positive and its results are likely to be more useful if you build evaluation in from the start and make it an on-going activity. For more information on this topic, click hereEvaluations fall into one of two broad categories: formative and summative. Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program. Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals.

Within the categories of formative and summative, there are different types of evaluation:

Type of EvaluationPurpose

Formative

1. Needs AssessmentDetermines who needs the program, how great the need is, and what can be done to best meet the need. An EE needs assessment can help determine what audiences are not currently served by programs and provide insight into what characteristics new programs should have to meet these audiences needs.

For more information, Needs Assessment Training uses a practical training module to lead you through a series of interactive pages about needs assessment.

2. Process or Implementation EvaluationExamines the process of implementing the program and determines whether the program is operating as planned. Can be done continuously or as a one-time assessment. Results are used to improve the program. A process evaluation of an EE program may focus on the number and type of participants reached and/or determining how satisfied these individuals are with the program.

Summative

1. Outcome EvaluationInvestigates to what extent the program is achieving its outcomes. These outcomes are the short-term and medium-term changes in program participants that result directly from the program. For example, EE outcome evaluations may examine improvements in participants knowledge, skills, attitudes, intentions, or behaviors.

2. Impact EvaluationDetermines any broader, longer-term changes that have occurred as a result of the program. These impacts are the net effects, typically on the entire school, community, organization, society, or environment. EE impact evaluations may focus on the educational, environmental quality, or human health impacts of EE programs.

For additional information on the differences between outcomes and impacts, including lists of potential EE outcomes and impacts, see MEERA's Outcomes and Impacts page.

Which of these evaluations is most appropriate depends on the stage of your program:

Adapted from Norland (2004), Pancer and Westhues (1989) and Rossi et al. (2004).

What makes a good evaluation?

Consider doing a best practices review of your program before proceeding with your evaluation.

If you are not clear on what you want to evaluate, consider first reviewing your program based on best practices such as the North American Association for Environmental Education (2004) Guidelines for Excellence Series:1. Nonformal Environmental Education Programs: Guidelines for Excellence

2. Environmental Education Materials: Guidelines for Excellence

3. Excellence in Environmental Education: Guidelines for Learning (preK-12)

4. Guidelines for the Preparation and Professional Development of Environmental Educators

You can conduct this review yourself or you can ask another environmental educator to help you. A best practices review is likely to identify program strengths and weaknesses, giving you important insight into what to focus your evaluation on.

A well-planned and carefully executed evaluation will reap more benefits for all stakeholders than an evaluation that is thrown together hastily and retrospectively. Though you may feel that you lack the time, resources, and expertise to carry out an evaluation, learning about evaluation early-on and planning carefully will help you navigate the process.

MEERA provides suggestions for all phases of an evaluation. But before you start, it will help to review the following characteristics of a good evaluation (list adapted from resource formerly available through the University of Sussex, Teaching and Learning Development Unit Evaluation Guidelines and John W. Evans' Short Course on Evaluation Basics):

Good evaluation is tailored to your program and builds on existing evaluation knowledge and resources. Your evaluation should be crafted to address the specific goals and objectives of your EE program. However, it is likely that other environmental educators have created and field-tested similar evaluation designs and instruments. Rather than starting from scratch, looking at what others have done can help you conduct a better evaluation. See MEERAs searchable database of EE evaluations to get started.

Good evaluation is inclusive. It ensures that diverse viewpoints are taken into account and that results are as complete and unbiased as possible. Input should be sought from all of those involved and affected by the evaluation such as students, parents, teachers, program staff, or community members. One way to ensure your evaluation is inclusive is by following the practice of participatory evaluation.

Good evaluation is honest. Evaluation results are likely to suggest that your program has strengths as well as limitations. Your evaluation should not be a simple declaration of program success or failure. Evidence that your EE program is not achieving all of its ambitious objectives can be hard to swallow, but it can also help you learn where to best put your limited resources.

Good evaluation is replicable and its methods are as rigorous as circumstances allow. A good evaluation is one that is likely to be replicable, meaning that someone else should be able to conduct the same evaluation and get the same results. The higher the quality of your evaluation design, its data collection methods and its data analysis, the more accurate its conclusions and the more confident others will be in its findings.

How do I make evaluation an integral part of my program?

Making evaluation an integral part of your program means evaluation is a part of everything you do. You design your program with evaluation in mind, collect data on an on-going basis, and use these data to continuously improve your program.

Developing and implementing such an evaluation system has many benefits including helping you to:

better understand your target audiences' needs and how to meet these needs

design objectives that are more achievable and measurable

monitor progress toward objectives more effectively and efficiently

learn more from evaluation

increase your program's productivity and effectiveness

To build and support an evaluation system:

Couple evaluation with strategic planning. As you set goals, objectives, and a desired vision of the future for your program, identify ways to measure these goals and objectives and how you might collect, analyze, and use this information. This process will help ensure that your objectives are measurable and that you are collecting information that you will use. Strategic planning is also a good time to create a list of questions you would like your evaluation to answer.

Revisit and update your evaluation plan and logic model (See Step 2) to make sure you are on track. Update these documents on a regular basis, adding new strategies, changing unsuccessful strategies, revising relationships in the model, and adding unforeseen impacts of an activity (EMI, 2004).

Build an evaluation culture by rewarding participation in evaluation, offering evaluation capacity building opportunities, providing funding for evaluation, communicating a convincing and unified purpose for evaluation, and celebrating evaluation successes.

The following resource provides more depth on integrating evaluation into program planning:

Best Practices Guide to Program Evaluation for Aquatic Educators (.pdf)Recreational Boating and Fishing Foundation. (2006).Beginner IntermediateChapter 2 of this guide, Create a climate for evaluation, gives advice on how to fully institutionalize evaluation into your organization. It describes features of an organizational culture, and explains how to build teamwork, administrative support and leadership for evaluation. It discusses the importance of developing organizational capacity for evaluation, linking evaluation to organizational planning and performance reviews, and unexpected benefits of evaluation to organizational culture.

If you want to learn more about how to institutionalize evaluation, check out the following resources on adaptive management. Adaptive management is an approach to conservation management that is based on learning from systematic, on-going monitoring and evaluation, and involves adapting and improving programs based on the findings from monitoring and evaluation.

Adaptive Management: A Tool for Conservation PractitionersSalafsky, N., R. Margoluis, and K. Redford, (2001) Biodiversity Support Program. BeginnerThis guide provides an overview of adaptive management, defines the approach, describes the conditions under which adaptive managements makes most sense, and outlines the steps involved.

Measures of Conservation Success: Designing, Managing, and Monitoring Conservation and Development ProjectsMargoluis, R., and N. Salafsky. (1998) Island Press.Beginner Intermediate AdvancedAvailable for purchase at Amazon.com.This book provides a detailed guide to project management and evaluation. The chapters and case studies describe the process step-by-step, from project conception to conclusion. The chapters on creating and implementing a monitoring plan, and on using the information obtained to modify the project are particularly useful.

How can I learn more?

Does your project make a difference? A guide to evaluating environmental education projects and programs. Sydney: Department of Environment and Conservation, Australia. (2004)BeginnerSection 1 provides a useful introduction to evaluation in EE. It defines evaluation, and explains why it is important and challenging, with quotes about the evaluation experiences of several environmental educators.

Designing Evaluation for Education Projects (.pdf),NOAA Office of Education and Sustainable Development. (2004)BeginnerIn Section 3, Why is evaluation important to project design and implementation? nine benefits of evaluation are listed, including, for example, the value of using evaluation results for public relations and outreach.

Evaluating EE in Schools: A Practical Guide for Teachers (.pdf),Bennett, D.B. (1984). UNESCO-UNEPBeginner IntermediateThe introduction of this guide explains four main benefits of evaluation in EE, including: 1) building greater support for your program, 2) improving your program, 3) advancing student learning, and 4) promoting better environmental outcomes.

Guidelines for Evaluating Non-Profit Communications Efforts (.pdf),Communications Consortium Media Center. (2004)Beginner IntermediateA section titled Overarching Evaluation Principles describes twelve principles of evaluation, such as the importance of being realistic about the potential impact of a project, and being aware of how values shape evaluation. Another noteworthy section, Acknowledging the Challenges of Evaluation, outlines nine substantial challenges, including the difficulty in assessing complicated changes in multiple levels of society (school, community, state, etc.). This resource focuses on evaluating public communications efforts, though most of the content is relevant to EE.

References:EMI (Ecosystem Management Initiative). (2004). Measuring Progress: An Evaluation Guide for Ecosystem and Community-Based Projects. School of Natural Resources and Environment, University of Michigan. Downloaded September 20, 2006 from: www.snre.umich.edu/ecomgt/evaluation/templates.htmPatton, M.Q. (1987). Qualitative Research Evaluation Methods. Thousand Oaks, CA: Sage Publishers.

Thomson, G. & Hoffman, J. (2003). Measuring the success of EE programs. Canadian Parks and Wilderness Society.

THE NATURE OF MONITORING AND EVALUATION

Definition and Purpose

by Phil Bartle, PhD

Workshop Handout

This explains what monitoring is and the purposes it serves

What is Monitoring?Monitoring is the regular observation and recording of activities taking place in a project or programme. It is a process of routinely gathering information on all aspects of the project.

To monitor is to check on how project activities are progressing. It is observation; systematic and purposeful observation.

Monitoring also involves giving feedback about the progress of the project to the donors, implementors and beneficiaries of the project.

Reporting enables the gathered information to be used in making decisions for improving project performance.

Purpose of Monitoring:Monitoring is very important in project planning and implementation.

It is like watching where you are going while riding a bicycle; you can adjust as you go along and ensure that you are on the right track.

Monitoring provides information that will be useful in:

Analysing the situation in the community and its project;

Determining whether the inputs in the project are well utilized;

Identifying problems facing the community or project and finding solutions;

Ensuring all activities are carried out properly by the right people and in time;

Using lessons from one project experience on to another; and

Determining whether the way the project was planned is the most appropriate way of solving the problem at hand.

Community Based Monitoring Workshop:

Copyright 1967, 1987, 2007 Phil BartleWeb Design by Lourdes Sada

Last update: 2011.09.30

Definitions, Meaning & characteristics of Planning.

Meaning and Concept of PlanningIn simple words, planning is deciding in advance what is to be done, when where, how and by whom it is to be done. Planning bridges the gap from where we are to where we want to go. It includes the selection of objectives, policies, procedures and programmes from among alternatives. A plan is a predetermined course of action to achieve a specified goal. It is an intellectual process characterized by thinking before doing. It is an attempt on the part of manager to anticipate the future in order to achieve better performance. Planning is the primary function of management.

Definitions of PlanningDifferent authors have given different definitions of planning from time to time. The main definitions of planning are as follows:

According to Alford and Beatt, "Planning is the thinking process, the organized foresight, the vision based on fact and experience that is required for intelligent action."

According to Theo Haimann, "Planning is deciding in advance what is to be done. When a manager plans, he projects a course of action for further attempting to achieve a consistent co-ordinate structure of operations aimed at the desired results.

According to Billy E. Goetz, "Planning is fundamentally choosing and a planning problem arises when an alternative course of action is discovered."

According to Koontz and O' Donnell, "Planning is an intellectual process, conscious determination of course of action, the basing of decision on purpose, facts and considered estimates."

According to Allen, "A plan is a trap laid to capture the future."

Nature / Characteristics of Planning

The main characteristics or nature of planning is given below:

Planning is an Intellectual ProcessPlanning is an intellectual process of thinking in advance. It is a process of deciding the future on the series of events to follow. Planning is a process where a number of steps are to be taken to decide the future course of action. Managers or executives have to consider various courses of action, achieve the desired goals, go in details of the pros and cons of every course of action and then finally decide what course of action may suit them best.

Planning Contributes to the ObjectivesPlanning contributes positively in attaining the objectives of the business enterprise. Since plans are there from the very first stage of operation, the management is able to handle every problem successfully. Plan try to set everything right. A purposeful, sound and effective planning process knows how and when to tackle a problem. This leads to success. Objectives thus are easily achieved.

Planning is a Primary Function of ManagementPlanning precedes other functions in the management process. Certainly, setting of goals to be achieved and lines of action to be followed precedes the organization, direction, supervision and control. No doubt, planning precedes other functions of management. It is primary requisite before other managerial functions step in. But all functions are inter-connected. It is mixed in all managerial functions but there too it gets precedence. It thus gets primary everywhere.

A continuous ProcessPlanning is a continuous process and a never ending activity of a manager in an enterprise based upon some assumptions which may or may not come true in the future. Therefore, the manager has to go on modifying revising and adjusting plans in the light of changing circumstances. According to George R. Terry, "Planning is a continuous process and there is no end to it. It involves continuous collection, evaluation and selection of data, and scientific investigation and analysis of the possible alternative courses of action and the selection of the best alternative.

Planning Pervades Managerial ActivitiesFrom primary of planning follows pervasiveness of planning. It is the function of every managerial personnel. The character, nature and scope of planning may change fro personnel to personnel but the planning as an action remains intact. According to Billy E. Goetz, "Plans cannot make an enterprise successful. Action is required, the enterprise must operate managerial planning seeks to achieve a consistent, coordinated structure of operations focused on desired trends. Without plans, action must become merely activity producing nothing but chaos."

Role, Significance, Importance & Advantages of Planning

An organisation without planning is like a sailboat minus its rudder. Without planning, organisation, are subject to the winds of organizational change. Planning is one of the most important and crucial functions of management. According to Koontz and O'Donnell, "Without planning business becomes random in nature and decisions become meaningless and adhoc choices." According to Geroge R. Terry, "Planning is the foundation of most successful actions of any enterprise." Planning becomes necessary due to the following reasons:

Reduction of UncertaintyFuture is always full of uncertainties. A business organisation has to function in these uncertainties. It can operate successfully if it is able to predict the uncertainties. Some of the uncertainties can be predicted by undertaking systematic. Some of the uncertainties can be predicted by undertaking systematic forecasting. Thus, planning helps in foreseeing uncertainties which may be caused by changes in technology, fashion and taste of people, government rules and regulations, etc.

Better Utilization of ResourcesAn important advantage of planning is that it makes effective and proper utilization of enterprise resources. It identifies all such available resources and makes optimum use of these resources.

Increases Organizational EffectivenessPlanning ensures organizational effectiveness. Effectiveness ensures that the organisation is in a position to achieve its objective due to increased efficiency of the organisation.

Reduces the Cost of PerformancePlanning assists in reducing the cost of performance. It includes the selection of only one course of action amongst the different courses of action that would yield the best results at minimum cost. It removes hesitancy, avoids crises and chaos, eliminates false steps and protects against improper deviations.

Concentration on ObjectivesIt is a basic characteristic of planning that it is related to the organizational objectives. All the operations are planned to achieve the organizational objectives. Planning facilitates the achievement of objectives by focusing attention on them. It requires the clear definition of objectives so that most appropriate alternative courses of action are chosen.

Helps in Co-ordinationGood plans unify the interdepartmental activity and clearly lay down the area of freedom in the development of various sub-plans. Various departments work in accordance with the overall plans of the organisation. Thus, there is harmony in the organisation, and duplication of efforts and conflict of jurisdiction are avoided.

Makes Control EffectivePlanning and control are inseparable in the sense that unplanned action cannot be controlled because control involves keeping activities on the predetermined course by rectifying deviations from plans. Planning helps control by furnishing standards of performance.

Encouragement to InnovationPlanning helps innovative and creative thinking among the managers because many new ideas come to the mind of a manager when he is planning. It creates a forward-looking attitude among the managers.

Increase in Competitive StrengthEffective planning gives a competitive edge to the enterprise over other enterprises that do not have planning or have ineffective planning. This is because planning may involve expansion of capacity, changes in work methods, changes in quality, anticipation of tastes and fashions of people and technological changes etc.

Delegation is FacilitatedA good plan always facilitates delegation of authority in a better way to subordinates.

Steps involved in Planning

Planning is a process which embraces a number of steps to be taken. Planning is an intellectual exercise and a conscious determination of courses of action. Therefore, it requires courses of action. The planning process is valid for one organisation and for one plan, may not be valid for other organizations or for all types of plans, because various factors that go into planning process may differ from organisation to organisation or from plan to plan. For example, planning process for a large organisation may not be the same for a small organisation. However, the major steps involved in the planning process of a major organisation or enterprise are as follows:

Establishing objectivesThe first and primary step in planning process is the establishment of planning objectives or goals. Definite objectives, in fact, speak categorically about what is to be done, where to place the initial emphasis and the things to be accomplished by the network of policies, procedures, budgets and programmes, the lack of which would invariably result in either faulty or ineffective planning.

It needs mentioning in this connection that objectives must be understandable and rational to make planning effective. Because the major objective, in all enterprise, needs be translated into derivative objective, accomplishment of enterprise objective needs a concrete endeavor of all the departments.

Establishment of Planning PremisesPlanning premises are assumptions about the future understanding of the expected situations. These are the conditions under which planning activities are to be undertaken. These premises may be internal or external. Internal premises are internal variables that affect the planning. These include organizational polices, various resources and the ability of the organisation to withstand the environmental pressure. External premises include all factors in task environment like political, social technological, competitors' plans and actions, government policies, market conditions. Both internal factors should be considered in formulating plans. At the top level mainly external premises are considered. As one moves downward, internal premises gain importance.

Determining Alternative CoursesThe next logical step in planning is to determine and evaluate alternative courses of action. It may be mentioned that there can hardly be any occasion when there are no alternatives. And it is most likely that alternatives properly assessed may prove worthy and meaningful. As a matter of fact, it is imperative that alternative courses of action must be developed before deciding upon the exact plan.

Evaluation of AlternativesHaving sought out the available alternatives along with their strong and weak points, planners are required to evaluate the alternatives giving due weight-age to various factors involved, for one alternative may appear to be most profitable involving heavy cash outlay whereas the other less profitable but involve least risk. Likewise, another course of action may be found contributing significantly to the company's long-range objectives although immediate expectations are likely to go unfulfilled.

Evidently, evaluation of alternative is a must to arrive at a decision. Otherwise, it would be difficult to choose the best course of action in the perspective of company needs and resources as well as objectives laid down.

Selecting a Course of ActionThe fifth step in planning is selecting a course of action from among alternatives. In fact, it is the point of decision-making-deciding upon the plan to be adopted for accomplishing the enterprise objectives.

Formulating Derivative PlansTo make any planning process complete the final step is to formulate derivative plans to give effect to and support the basic plan. For example, if Indian Airlines decide to run Jumbo Jets between Delhi an Patna, obliviously, a number of derivative plans have to be framed to support the decision, e.g., a staffing plan, operating plans for fuelling, maintenance, stores purchase, etc. In other words, plans do not accomplish themselves. They require to be broken down into supporting plans. Each manager and department of the organisation is to contribute to the accomplishment of the master plan on the basis of the derivative plans.

Establishing Sequence of ActivitiesTiming an sequence of activities are determined after formulating basic and derivative plans, so that plans may be put into action. Timing is an essential consideration in planning. It gives practical shape and concrete form to the programmes. The starting and finishing times are fixed for each piece of work, so as to indicate when the within what time that work is to be commenced and completed. Bad timing of programmes results in their failure. To maintain a symmetry of performance and a smooth flow of work, the sequence of operation shaped be arranged carefully by giving priorities to some work in preference to others. Under sequence it should be decided as to who will don what and at what time.

Feedback or Follow-up ActionFormulating plans and chalking out of programmes are not sufficient, unless follow-up action is provided to see that plans so prepared and programmes chalked out are being carried out in accordance with the plan and to see whether these are not kept in cold storage. It is also required to see whether the plan is working well in the present situation. If conditions have changed, the plan current plan has become outdated or inoperative it should be replaced by another plan. A regular follow-up is necessary and desirable from effective implementation and accomplishment of tasks assigned.

The plan should be communicated to all persons concerned in the organisation. Its objectives and course of action must be clearly defined leaving no ambiguity in the minds of those who are responsible for its execution. Planning is effective only when the persons involved work in a team spirit and all are committed to the objectives, policies, programmes, strategies envisaged in the plan.

Dr. Michaela Zint, Project Leader

Dr. Michaela Zint, Project Leader

Dr. Michaela Zint, Project Leader