Evaluation Best Practices

Results Based & Logic Model ProgramsResults Based & Logic Model Programs

Bob Craven, Director – Digital CenterBob Craven, Director – Digital CenterBrad Lubben, Director – North Central RegionBrad Lubben, Director – North Central RegionJo Ann Warner – Assistant Director – Western RegionJo Ann Warner – Assistant Director – Western RegionKevin Klair – Digital CenterKevin Klair – Digital Center

With help from: With help from: Kynda Curtis, Utah State UniversityKynda Curtis, Utah State UniversityGary Hachfeld, University of Minnesota ExtensionGary Hachfeld, University of Minnesota Extension

The Extension Risk Management Education Grants Program Team

Our Mission

“Helping America’s farmers and ranchers manage risk”

Extension Risk Management Education

Program University of Delaware – Northeast Region

University of Arkansas – Southern Region

University of Nebraska - North Central Region

Washington State University Extension – Western Region

University of Minnesota – Digital Center

USDA, National Institute of Food and Agriculture

Regional Centers North Central Center

University of Nebraska

Northeast Center University of Delaware Cooperative Extension

Southern Center University of Arkansas Division of Agriculture

Western Center Washington State University Extension

Return on Investment

• The return on investment of Extension RME awarded grant dollars are the risk management improvements that agricultural producers make as result of a project.

• The risk management results/outcomes that agricultural producers achieve “are” the measure of success of any project we fund.

Process Focused on Results/Outcomes

• The results based process, including the grant selection, reporting, and management system,

– increases the probability that funded programs will improve the risk management skills of the farmers & ranchers who participate,

– is designed to improve, over time, the capacity and quality of the programs we fund, and

– increases the opportunities to improve the economic viability of the farm and ranch family participants.

Evaluation Best Practices Keep the focus on:

Producer Results/Outcomes

Project Team’s Accountability to Measuring/Verifying Producer Results

Commitment to helping producers reach their risk management goals

Ability to Measure Outcomes

• Improve effectiveness by looking at your project from start to finish in terms of results for participants.

• Demonstrate participant achievement of risk management tools and strategies by posting on the public reporting system (final reports on ExtensionRME.org & the Ag Risk Education Library)

• Move the bar higher for the effectiveness of risk management education

Beginning with the “end” in mind• Begin with the end in mind

– Determine firstfirst what the result/outcome is that the participants will achieve.

– Design the program to lead the participants to those results/outcomes.

• Do not wait till the end of the project to determine what the outcomes might be.

Outcomes- Will Depend Upon:

- Knowledge of your target audience

- Why ag producers will choose to participate

- Action and learning processes (project steps) that help producers achieve long term outcomes

- Program resources including the team, collaborators and expert presenters sufficient to lead participants to the desired outcomes.

Outcomes – A Systems Approach

The Proposal as a working document

i. Application Stage

ii. Program Delivery

iii. Reporting

Outcomes & the Logic Model

Identifying and Articulating Program Outcomes

Participant Outcomes

“Begin with the end in mind” Or

“plan backward, implement forward”

Jon Newkirk, Director, WSU Western Center Beverly Anderson Parsons, WKKF Cluster

Evaluator

An Outcome Defined….

Change in:KnowledgeActions orConditions

Outcomes Defined….

Changes in knowledge…When project participants 1. Learn or become aware of new

fundamental or applied knowledge which could include:a) Methods and techniquesb) Policy Knowledgec) Improved skills such as decision making, life

skills or quality of life choices

Outcomes Defined….Change in Actions…

When project participants 1. Act upon what they have learned such as:

a) Implementing methods and techniquesb) Changing a practicec) Adoption of improved skills such as decision

making, life skills or quality of life choicesd) Direct application of knowledge gained

Outcomes Defined….

Change in Conditions…A socio/economic condition is positively changed because of participant actions1. Information resources that improve

community and/or farming infrastructure2. Technology transfer3. Improved farm income for target audience4. Decrease in farm foreclosures

A Logic Model can help you if you begin with the end in mind….

A Logic Model can help to identify your A Logic Model can help to identify your outcomes (changes in knowledge, outcomes (changes in knowledge,

actions or conditions) and the actions or conditions) and the activities and partners that will activities and partners that will support the accomplishment of support the accomplishment of

outcomes for project participantsoutcomes for project participants

Logic Model as a Tool to Construct Logic Model as a Tool to Construct Realistic OutcomesRealistic Outcomes

FirstFirst – Identify the situation(s) that your project will help to address (farm foreclosures, high price of inputs, lack of record keeping skills etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer Demand).

1.1. OutcomesOutcomes – The identification of changes in knowledge, actions, or conditions for program participants

2.2. OutputsOutputs – (activities – curriculum development, workshops, trainings etc.) that will support and lead to the achievement of participant outcomes

3.3. InputsInputs – Resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc.

Logic Model Beginning With the End in MindLogic Model Beginning With the End in Mind

OUTCOMES

(Identify Measurable

Risk Management

Results)

OUTPUTS

(Lead producer participants to the

desired risk management

results)

INPUTS

(Resources that will strengthen

producer achievement of

risk management results)

INPUTSINPUTSOUTPUTSOUTPUTS

OUTCOMESOUTCOMES

Logic Model Implementing Logic Model Implementing ForwardForward

Producer Results Outcomes Impacts

(Learning – Action – Condition )

A Hierarchy of Actions and Decisions that Range from Improved Understanding to Incorporating Beneficial Risk Management Practices

Producer Results/Outcomes – an example of progressive steps…

↓ UnderstandUnderstand or have improved understanding of the risk or have improved understanding of the risk issue, tool or need to take action.issue, tool or need to take action.

↓ DecideDecide to take an action. Producer commitment to write a to take an action. Producer commitment to write a business or marketing plan, set up a record keeping system, business or marketing plan, set up a record keeping system, conduct a risk analysis, or initiate other steps that will lead to conduct a risk analysis, or initiate other steps that will lead to improved risk management strategiesimproved risk management strategies

↓ DevelopDevelop business tools to strengthen economic capacity, business tools to strengthen economic capacity, such as writing a marketing plan, business plan, farm transfer such as writing a marketing plan, business plan, farm transfer plan, or performing a risk assessment etc.plan, or performing a risk assessment etc.

ImplementImplement or take an action that improves or optimizes their or take an action that improves or optimizes their profitability– implementing a marketing plan, using a record profitability– implementing a marketing plan, using a record keeping system etc.keeping system etc.

In Summary…Results/OutcomesResults/Outcomes

– Measure of success for the participant• Ability to change/incorporate/implement/adopt risk

management practices as a result of participation in your project.

– Achievable and measureable within a reasonable timeframe during and following training.

– Sequential• Identifies changes in condition or practice you seek

for the participant• Specifies the degree of change required for

participant success• Stated in a measurable way knowing methods you

will use to verify changes in condition, practice etc.

Optimizing Producers’ Chances for Success

FirstFirst – Identify the situation(s) that your project will help to address (farm foreclosures, high price of inputs, lack of record keeping skills etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer Demand).

Then…Identify the Then…Identify the Results/OutcomesResults/Outcomes – changes in – changes in knowledge, actions, or conditions for program participantsknowledge, actions, or conditions for program participants..

Describe what outputsoutputs (activities – curriculum development, workshops, trainings etc.) will support and lead to the achievement of participant outcomes. Create a realistic timetable for your actions (Project Steps)Create a realistic timetable for your actions (Project Steps)

Describe inputsinputs – resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc.

Logic Model Beginning With the End in MindLogic Model Beginning With the End in Mind

OUTCOMES

(Identify Measurable

Risk Management

Results)

OUTPUTS

(Lead producer participants to the

desired risk management

results)

INPUTS

(Resources that will strengthen

producer achievement of

risk management results)

INPUTSINPUTSOUTPUTSOUTPUTS

OUTCOMESOUTCOMES

Logic Model Implementing Logic Model Implementing ForwardForward

Using the Logic Model to Identify Results/Outcomes & Create a Plan

of Work (Project Steps)

Planning Backwards – Beginning with the End in Mind

Results/Outcomes Activities/Tasks Responsible Entities & Roles

Outcomes Outputs Inputs

Implementing Forward Responsible Entities & Roles Activities/Tasks Results/Outcomes Inputs Outputs Outcomes

Post/Short Term Follow-up

Long Term or 3-6 Month Follow-up

Project Steps

Production Risk ManagementOutputs or Outcomes?• A series of insurance workshops will

be delivered to producers• A crop/livestock insurance

spreadsheet will be developed• Producers will increase purchases of

crop/livestock insurance• Producers will evaluate crop/livestock

insurance decisions

Production Risk ManagementOutputs or Outcomes?• A series of insurance workshops will be

delivered to producers - output• A crop/livestock insurance spreadsheet

will be developed - output• Producers will increase purchases of

crop/livestock insurance – w. outcome• Producers will evaluate crop/livestock

insurance decisions – s. outcome

Marketing Risk ManagementOutputs or Outcomes?• Producers will attend a marketing

workshop• A marketing curriculum will be

written• Producers will implement a

marketing plan• Producers will write a marketing

plan

Marketing Risk ManagementOutputs or Outcomes?• Producers will attend a marketing

workshop – output• A marketing curriculum will be

written - output• Producers will implement a

marketing plan – s. outcome• Producers will write a marketing

plan – w. outcome

Financial Risk ManagementOutputs or Outcomes?

• Producers will calculate financial ratios for case farms/ranches

• Producers will understand and interpret financial ratios

• Producers will evaluate the financial ratios for their farm/ranch

Financial Risk ManagementOutputs or Outcomes?

• Producers will calculate financial ratios for case farms/ranches - output

• Producers will understand and interpret financial ratios – w. outcome

• Producers will evaluate the financial ratios for their farm/ranch – s. outcome

Legal Risk ManagementOutputs or Outcomes?

• Producers will attend a farm program educational meeting

• Producers will successfully analyze farm program sign-up decisions

• Farm program fact sheets and calculators will be produced

Legal Risk ManagementOutputs or Outcomes?

• Producers will attend a farm program educational meeting - output

• Producers will successfully analyze farm program sign-up decisions – s. outcome

• Farm program fact sheets and calculators will be produced - output

Human Resources Risk ManagementOutputs or Outcomes?

• Succession planning materials will be published

• A series of programs will discuss the components of a farm succession plan

• Producers will develop a succession plan

• Producers will implement a succession plan

Human Resources Risk ManagementOutputs or Outcomes?

• Succession planning materials will be published - output

• A series of programs will discuss the components of a farm succession plan - output

• Producers will develop a succession plan – w. outcome

• Producers will implement a succession plan – s. outcome

Exercise – Writing Effective Outcomes

1. Identify the risk situation(s) and the risk topic areas that you want your project to address

2.2. Build a set of Build a set of measurablemeasurable risk management risk management results/outcomesresults/outcomes

Include both short and long term outcomes (results that will happen after the project – within 3 months to a year)

3. Determine if the outcomes are a good fit for the target audience

*Handout

Target AudienceAre Outcomes suitable for audience being

served?Realistic goals that can be achieved within specified timeframe?Linguistically and Culturally Appropriate?

Learning Styles and Environment?

Reliability of InformationScience based content?Appropriate for outcomes being proposed?

How well do you know your target audience?Relationship Capital?

Evaluation MethodologyEvaluation Methodology

Measuring OutcomesMeasuring Outcomes

Evaluation Mechanics Protocols for Administering Evaluations

– Preparing Workshop Facilitators• Evaluation Procedures Audience Preparation

Methodologies– Pre/Post/Follow-Up– Ease of design and implementation

• Methods suitable to audience• Framing questions appropriately to know if outcomes have been

achieved. • Questions designed to measure outcomes should flow/follow from

the hierarchy of results/outcomes.• Qualitative and Quantitative

– Open and close ended questions

Gathering informal data- Producer Stories and Testimonials

Evaluation Mechanics Tabulating the results

Who will collect the data?How will data be collected?When will data be collected?What will be done with the compiled data?

These questions need to be addressed before administering your evaluations and are an integral part of program design!

Evaluation Mechanics

Suitable Question Formats– Multiple Choice – Yes/No – Rating scales– Open ended

Simple for respondent to understandEasier to analyze for project teams

with less evaluation experience

Evaluation Mechanics

Multiple Choice – Yes/No1. Assign a numeric value to each

question response For example code “yes” responses as “1”

and “no” responses as “2”)

2. Use the “count” function in Excel to analyze the responses Pie Charts can be used to graphically depict

responses

Evaluation MechanicsRating Scales

– Allow respondents to indicate level of agreement with a statement or to rate level of importance.

Straightforward and easy to understandEasier to analyze than other question formats (because responses are already in numeric form, it is not necessary to recode them)Shows how respondents feel on average about the question being asked.

Evaluation MechanicsOpen ended questions

– Invite more detail from the respondent– Require a little more effort from

respondentCan identify challenges/barriers to progressProvide a complete story or testimonial

– Analysis may require identifying common themes

Evaluation Mechanics– Add yes’s and no’s– Add up ranking #’s– Provide an average of answers,

rankings, or range– Organize comments into similar patterns – Report results should be structured to

show that participants goals/outcomes are being achieved.

• Numbers first, percentages second

Keep it simple!

NIFA Metrics for Measuring Outcomes

Testing your methods - Will your evaluation tool(s) clearly

demonstrate progress participants are making towards achieving the outcomes?- Changes in Knowledge?- Changes in Attitude?- Changes in Behavior/Approach?- Changes in Economic or Social

Circumstances?

Pre Course Evaluation

Purpose

1.Benchmark info on knowledge levels2.Information on participant background,

past experience, attitudes and needs for program managers and instructors

Combining Pre/Post into Post• “Please rate your knowledge PRIOR

to attending today’s workshop”• “Please rate your knowledge AFTER

attending this workshop”• “Please rate to what extent you have

the knowledge, tools and resources needed to …”

• “I better understand”Likert Scale or Similar Method

Post Course Evaluation

Purpose:

MeasuresKnowledge GainAttitude ChangeActions participants expected to take as a

result of attending course

Post/Short Term Follow-up

Follow-Up Evaluation

Purpose:

To see what actions participants have taken as a result of attending your course and how these actions have led to economic and other benefits.

Long Term or 3-6 Month Follow-up

3 – 6 Month Follow-Up

3 – 6 Month Follow-Up Cont.

Ease of Design

Do evaluation questions flow logically from sequentially stated outcomes?

Asked in a manner to obtain specific information on level of accomplishment or barriers encountered?

Build on progress/commitments made from short term or post evaluations?

Usability of information?

Ease of Design

Avoid “respondent fatigue”?

Use definitive language to obtain accurate responses? Avoid words like “generally” and “usually” – i.e. rather

than “Do you generally follow a business plan?”, “Do you follow a business plan?”

Keep it short and simple!

Exercise – Writing an Effective Evaluation

Post – Short TermFollow-Up – Long Term

Within 3 months to a year

Evaluation Tools

Variety of Methods:– Pre/Post Tests

• Audience Response Systems (Clickers)

– Follow-up Evaluations (3 Months – Year)• Mail or Internet Survey

– Telephone Interview– Face-to-face interviews (in-person survey)– Dot survey– Informal Interview

Evaluation Tools

Considerations:

Time FrameBudgetAudience characteristics

Results/Outcome Based Reporting

• Documents:– Producer Progress towards achieving

goals (short and long term)– Producer Stories and Testimonials– Unexpected results/outcomes

• Provides Access To:– RME Curriculum and resource materials

• Training methodologies• Evaluation Tools

Integrating Results/Outcomes into

Project Reporting Summarizing Outcomes

Project Overview Actual Results/Outcomes Comments/Stories/Examples of Success

Input into other reports

Impactful Reporting• Builds capacity for educators who work directly with farmers

and ranchers through… • Access to risk management curriculum and resources• Access to training methodologies• Access to successful evaluation methods and tools

• Creates transparency• Improves accountability• Documents in a database format risk management

outcomes/impacts for farmers and ranchers.• Available to federal and state agencies, government offices such as the GAO and

OMB, institutions and other funding partners.

• Keeps moving the bar higher for the effectiveness of risk management education.

•

Evaluation Best PracticesMeasuring results/outcomes Measuring results/outcomes

successfully:successfully:

• Choose methods that balance best practices with available resources – Combination of direct and indirect measures

• Direct measures (assignments, action plans etc.) yield stronger evidence of change in knowledge and beneficial practices, but are more time intensive and may take more expertise to evaluate.

• Indirect measures can give instant feedback but may not provide real evidence of knowledge gained.

• Qualitative measures (like phone interviews or focus groups) that may be more resource intensive could be accomplished through a representative sample.

Evaluation Best Practices

Measuring results/outcomes Measuring results/outcomes successfullysuccessfully

• A randomly sampled group, representing approximately 5 percent of participants submitting a post-meeting evaluation, will be surveyed by phone. Participants will be asked whether or not they have completed and implemented their farm/ranch transition and personal estate plan. They will be asked what challenges and obstacles they encountered during the process, how the program assisted them in the process and how useful the resource workbook was to them.

Evaluation Best Practices

Measuring results/outcomes successfully

– Direct Interactions with Producers can increase the chance that participants will implement beneficial risk management practices.

– Good follow-up can lead to collegial relationships that may bring results beyond the initial results/outcomes that were initially projected (Unexpected results).

– Good program design allows for learning and application to take place in linked stages within a timeframe that encourages the implementation/adoption of longer term results/outcomes.

Evaluation Best Practices

• Interactive discussion on best practices – Successes – Pitfalls

*Handout

In Summary

If the “Return on Investment” of grant dollars awarded are the risk management improvements that producers will make…

Then measuring producer results (outcomes) becomes an integral component of program planning and design, not an afterthought!

Thank You!Bob Craven, Director – Digital Center, University of MinnesotaBob Craven, Director – Digital Center, University of Minnesotarcraven@umn.edurcraven@umn.eduBrad Lubben, Director – North Central Region, University of Nebraska-LincolnBrad Lubben, Director – North Central Region, University of Nebraska-Lincolnblubben2@unl.edublubben2@unl.eduJo Ann Warner , Assistant Director – Western Region, Washington State UniversityJo Ann Warner , Assistant Director – Western Region, Washington State Universitywarnerj@wsu.eduwarnerj@wsu.eduKevin Klair – Digital Center, University of MinnesotaKevin Klair – Digital Center, University of Minnesotakklair@umn.edukklair@umn.edu

Kynda Curtis, Utah State UniversityKynda Curtis, Utah State Universitykynda.curtis@usu.edukynda.curtis@usu.eduGary Hachfeld, University of Minnesota ExtensionGary Hachfeld, University of Minnesota Extensionhachf002@umn.eduhachf002@umn.edu

Results/Outcome ResourcesThe Rensselaerville Institute:http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266

“Outcome Funding: A New Approach to Targeted Grantmaking”

Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key questions that the funder as investor should ask: What am I buying? What is the probability I will get it? Outcome Funding begins with an abrupt and timely challenge to the standard wisdom of the Request for Proposal process. The authors make the point that very little of the traditional proposal content addresses results, favoring explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical on how to achieve it. 

Logic Model Templates:http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html

Developing a Basic Logic Model For Your Program:

http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf I have become convinced that it makes a considerable difference if you do the outcomes before planning the

activities. I definitely advocate doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, “plan backward, implement forward.” Beverly Anderson Parsons, WKKF Cluster Evaluator