How to Increase Extension Program Outcomes and Impacts (Slides for Educators)
-
Upload
dr-ayanava-majumdar-auburn-university -
Category
Education
-
view
262 -
download
3
description
Transcript of How to Increase Extension Program Outcomes and Impacts (Slides for Educators)
How to Improve Extension Program Outcomes using a Monitoring and Evaluation
Strategy
Dr. Ayanava MajumdarExtension Entomologist, SARE Coord. (AU)
111-A Duncan Hall, Auburn [email protected]
251-331-8416
Focus of this presentation• Structure & implementation of Alabama IPM
Program• Provide some basics of evaluation – Extension
perspective!• Encourage you to be consistent in your
evaluation efforts• Develop capacity building within organization
About the Author (Ayanava)Ext. evaluations:
• Memberships: American Eval. Assoc., Southeast Eval. Assoc.
• AEA365 Blog Curator (2010-2011), Organization Capacity Building TIG Grant Panel, Ext. Eval. TIG member
• Lead evaluator: USDA-NIFA, AL DoA, commodity grants, School IPM, AACAAS, NACAA
Organizational capacity building:
• Initiated ACES Program Evaluation Resource Committee, 2010… created the Evaluation Toolkit [online]
• Workshops, webinars, publications: total ~7 hr, trained 161 Agents, County Coordinators, Specialists, Ext. Administrators
Alabama Small Farms• Produce fruits & vegetables that
are consumed locally
• Area = 6,000+ acres
• Crop value = $48 million
• Growth rate for industry = 14%
• Vegetables: Tomatoes, sweet corn, watermelon, crucifers
• Farmer markets: 135 (1100 farms)
• Direct sales from AL farmers to consumers account for 0.2% of farm sales (Meter 2012)
Alabama Cooperative Extension System
Ref.: Robinson, Dubois, Bailey (2005). Journal of Extension.
Regional Extension Agents (REAs)
Regional or statewide IPM events
Outputs change annuallyOutcomes do not change!Impacts now underway.
+ CECs
ACES Commercial Horticulture Team – Vegetable IPM Team Members
ACES Home Grounds Team – Vegetable IPM Team Members
Chris Becker, REA Willie Datcher, REA Mike McQueen, REA
Alfred Jackson, Tuskegee Extension
Awards & Recognition• ‘Friends of IPM - Pulling Together Award’ from the Southern
Region IPM Center, Raleigh, NC (2014) for IPM newsletter.• Friends of IPM: Future Leader Award (2012)
By Southern Region IPM Center, Raleigh, NCRecognizes excellence in IPM program directly
• Search for Excellence – Crops (2012)By National Association of County Agricultural AgentsRecognizes a high impact IPM program
• Achievement Award (2012)By the National Association of County Agricultural Agents
• Communication Awards (6) By National Association of County Agricultural Agents
Structure of Alabama IPM Program:A Model Program for Study
Fund acquisition (external)• Funds needed to support REA travel & IPM
demonstration plots
• USDA/NIFA Collaborative Grants: $124,500 (part of two large grants worth $2.1 million) = 6%
• SARE support: $10,000 + personnel for website
• Alabama Department of Agriculture: $25,000
• Industry support: $45,000
Theory of Planned Behavior (TpB)
Ajzen, I. 1991. The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes. 50: 179-211
Theory of Change: Ext. Logic Model
IPM Theory of Change
Communities
ImpactsModified from Kokate et al. 2009
Farmer-to-farmerTrain-the-trainer
Sustain change
Key farmers
Technology acceptance
ExtensionResearchProject Management
Capacity Building
Need identified(context)
Ext. IPM Project Implementation
Transformational
learning
(full tech.
adoption)
Information
transfer
(tech.
awareness)
Service
(technology
integration &
refinement)
Facilitation
(demonstration/
tech. transfer)
Keep informed(state ag dept)
Cooperate closely(key farmers) Network closely
(commodity group)
Collaborate(industry)
Continuous needs & outcomes assessments
Transformational education model Stakeholder expectation
1
2
3
4
1
2
3
4
Model by Dr. A. Majumdar 2011Based on Rockwell et al. 2003
Co
mm
un
icat
ion
Str
ateg
y (I
PM
-CO
RE
)
No cost tools
Technology identified & assessed
Story developed
Press releaseExtension Websites Social sharing
(Slideshare, YouTube)Social networking (Facebook)
Medium cost tools
IPM Communicator Newsletter
Extension regional meetings, IPM
Web Conferences
IPM Project flyers,
bookmarksHigh cost tools
Multiple partnerships (~20)
IPM Exhibitions,Newspaper
advertisements
IPM on-farm demonstrations/communications
Small Farm IPM Field
Guide
IPM App?
Level 1: Regional IPM Meetings• Participants: Vegetable producers, certified
organic farms, gardeners & homeowners, crop advisors, nonprofit organizations (partners)
• Teaching techniques: Presentation, pest specimens
• Typical length of talks: 46 min• Participants total: 1,535• Number of events completed: 58• Evaluation: Events are monitored for quality
and impacts via multimodal surveys
Regional IPM Training Meetings (Level 1 training)
Level 2: IPM Workshops
• Participants: Small vegetable producers, low resource farmers, Master Gardeners, nonprofit organizations, Extension Agents (in-service)
• Teaching techniques: Case studies, simulations, specimens
• Typical length of talks: 140 min• Participants total: 570 • Number of events completed: 14• Evaluation: Qualitative & quantitative feedback,
impact evaluations
Hands-on IPM Workshops (Level 2 training)
Level 3: IPM Field Days (Demonstration Sites)
• Participants: Producers, low resource farmers, gardeners
• Teaching techniques: Scouting, pest identification, decision-making system
• Typical training time: 3-4 hours• Participants total: 308• Number of events completed: 12• Evaluation: Qualitative & quantitative feedback,
impact evaluations, farm visits
IPM Field Days (Level 3 training)
Farmer-to-farmer training model is very influential for IPM project
Farmer-to-Farmer training during IPM Field Days
Causes high information retention & impact on communities
Basics of Program Monitoring & Evaluation (PME)
Evaluation• Evaluation is the systematic collection of information
about the merit, worth or significance of a program (Scriven 1999).
• Evaluation is broadly defines as the systematic collection of data – both quantitative and qualitative – to aid users in developing knowledge about & managing a targeted set of activities (Scheirer 2012).
• Ext. evaluation practice aims to find plausible conclusions, not cause & effect!
Rate your IPM program evaluation experience(useful exercise for audience)
Common myths!
• My clients like me, so I do not need to evaluate.
• Evaluation is hard & outside my job description.
• Public does not respond to my surveys.
• Nobody is going to look at the evaluations once they are done.
• Only ‘experts’ can do evaluation. Why bother!
PERCEPTION REALITY• Evaluation helps document your
success.
• 20% effort generates 80% information.
• Provide time to respond, part of agenda, show improvements!
• Share the results with all stakeholders.
• Evaluation can be done by any trained individual committed to the standards.
Who can do the evaluation?Internal evaluators
• Project leadership team directly guides evaluation (cost-effective)
• Indicates organizational commitment /capacity building
• Project monitoring possible (utilization-focus)
• Fast assessment of reactive programs
• Good participation
External evaluators
• Neutral third-party observers (expensive)
• No internal capacity building
• Good for impacts
• Slow assessments & less useful
• Poor participation (trust)
Based on Patton (1997)
CDC Framework for Program Evaluation Standards
Thomas Chapel, Chief Eval. Officer, CDC
Purpose of Evaluation
Formative evaluation• Done when the program
is active• Project monitoring• Continuous use• Flexible (mixed) designs
Summative evaluation• Done at the end of a
program• Measure impacts• One time use• Rigid design, expensive
Patton (1997), Scriven (1972)
Utilization-focused Evaluations• Evaluation technique emphasizing USE (Patton 1993)
• Improvement-oriented evaluation: includes formative evaluations, quality enhancement, Total Quality Management (TQM)
• TQM includes using information systems to monitor program efforts & outcomes continuously
• Use feedback as a monitoring system (like insect traps!).
+ Trouble= = Do something about it!
Evaluation techniquesReactive assessment:• Participants are aware of the
assessment• Examples: surveys, interviews,
tests
Non-reactive assessment:• Participants are unaware of the
assessment• Examples: observation
Haas (2005)
Choose outcome ‘indicators’ carefully
• Indicators are critical questions• Based on program objectives• Used for monitoring progress• Usually expressed as rates, percentages,
efficiencies, etc.• There should be few trackable indicators
(QUAL and QUAN)
Taylor-Powell & Henert 2008
Needsassessments
Processevaluations(formative)
OutcomeEvaluations(formative)
Impactevaluations
(summative)
Project monitoring
Project improvement
Modified strategy
Modified outputs
Evaluation Strategy in Extension IPM Program
Graphic by Dr. Ayanava Majumdar, ACES
What do we measure?Hierarchy of Outcome Effects
• Level 1: REACTION • Level 2: LEARNING
• Level 3: BEHAVIOR• Level 4: RESULT
Increasing complexity of
evaluation
Evaluate during program
Kirkpatrick Four Levels, 1959, 1994
Post program evaluation
QUAN
QUAL
Hierarchy of Effects in IPM Program (In Pictures)
• Level 1: REACTION
• Level 2: LEARNING
Hierarchy of Effects in IPM Program (In Pictures)
• Level 3: BEHAVIOR
• Level 4: RESULTS
AL IPM ProjectOUTCOMES
(Usefulness of Project Monitoring)
Project outcomes (2008-2012)• Developed a unique Theory of Change and
Transformational Extension Education Model based on long-term outcomes
• New producers reached 14% and rising (transitioning farms 7-10%).
• General IPM adoption rate: over 90%• Adoption rate for unique IPM tactics: 63% (up from
38%)• Publication in print & electronic have significantly
reduced barrier to IPM adoption • Funding for Small Farm IPM program has doubled
since 2012. Much focus on Ext. Agent training.
• Rising interest of producers/Pull on project:– Over 10 times rise in participation in IPM events– 10 times rise in number of IPM events
• Barriers to IPM adoption (crop advisors):– Lack of awareness reduced by 33% in 3 years– Difficulty in accessing information reduced by 33%
• Barriers to IPM adoption (producers):– Lack of awareness reduced 16% in 3 years– Difficulty in accessing IPM information reduced 13%
Project outcomes (2008-2012)
IPM Theory of Change
Communities
ImpactsModified from Kokate et al. 2009
Farmer-to-farmerTrain-the-trainer
Sustain change
Key farmers
Technology acceptance
ExtensionResearchProject Management
Capacity Building
Need identified(context)
2009 2010 2011 20120
100200300400500600700800900
1000
67
259
505
945
39 54 57 553 4 28 27
Total participants
Survey return rate (%)
Major IPM Events
GROWING INTEREST OF FARMERS IN IPM
2009 2010 20110
10
20
30
40
50
60
70
80
1.56 6.9
73
36
26
58 10.18
3429.1
117
20.9
Beginning farmers (%)Conventional producers (%)LRFs (%)Transitioning farms (%)Gardeners (%)
NATURE OF AUDIENCE AT IPM EVENTS
2009 2010 20110
5
10
15
20
25
30 28
13.6
9.3
Average land holding (acres)
CHANGING NATURE OF EXTENSION CLIENTELE(reason for splitting IPM program into two campaigns to
increase impacts)
Organic/Small Farms IPM
Campaign
Conventional farmers(with 15 to 50 acres)
2012
Farmers(<10 acres)
Gardeners(<2 acres)
Conventional/Large Farms IPM
Campaign
Conventional vegetable farmers = 15-50 acres
Transitioning farmers = 10-14 acres
Beginning farmers = <10 acresLow resource farmers = 2 to 4 acres
CHANGING NATURE OF EXTENSION CLIENTELE(reason for splitting IPM program into two campaigns to
increase impacts)
Two IPM Campaigns launched in 2012 with separate strategies and outcomes. Rapid increase in small
farm IPM funding and number of participants due to rising interest in local food systems.
2008 20110
5
10
15
20
25
30
35
40
45
5045
29
10
23.2
30
2.2
20
11.4
2.55
Lack of awareness (%)
High cost (%)
Difficult to access information (%)
Low availability of insecticides (%)
Lack of time (%)
BARRIERS TO IPM ADOPTION & CORRECTIONS
Major concerns at the time
PROJECT IMPACTS(Watch 2013 Impact Video at http://
youtu.be/aqrjQINLUdw)
Impact Evaluation, 2010-2011 (n=58):
• E-subscriptions growth rate = 12%
• Current subscription base ~1350 with much higher readership.
• 34% farmers, 28% company representatives, 38% others
• 53% read for 15 minutes and 22% for 30+ minutes
• 94% support continuation of the newsletter (2011 survey)
• 70% use the information (2013 survey)
• Six financial gains reports (2013): $3,550 in pesticide saving
• Short-term impact of publication: $591 per case of adoption
IPM Newsletter Impacts
Program Impacts: IPM AdoptionIndicator 2011 2010 2009 Increase
Adoption of insect monitoring/scouting practices
73.5% 41.0% +32%
Adoption of insecticide recommendations
79.6% 46.5% 33%
Use of biological insecticides
42.3% 26.5% 16%
Overall IPM adoption 62.7% 38.5% 24%
Economic & social impact assessments ongoing in 2012 and beyond.
Received IRB Approval for Surveys.
Economic impacts• Yield loss in the absence of IPM: 44% or
more• Vegetable producers gained $246 per acre by
using IPM recommendations• Profits occur by judicious use of pesticides,
adoption of appropriate control tactics after pest identification, and timely action.
• Weekly IPM newsletter resulted in saving of over $500 per adoptive farmer
• Increased use of biological insecticides (42%)
• Conservation of natural enemies (unknown)
• Profits occur by judicious use of pesticides, adoption of appropriate control tactics after pest identification, and timely action.
Environmental impacts
• Three new partnerships with influential commodity & consumer organizations
• New partnerships with nonprofit agencies serving LRFs in Black Belt of Alabama
• Impact assessments continue in 2012 & beyond.
Social impacts
Charles Brannon, vegetable producer (Addison, AL): “This is the first year we used the IPM recommendation from the handbook and received training from Dr. A. In the hoop house alone, we sold about $5,500 worth of tomatoes which is double the output from last year. Fruitworms and stinkbugs used to do about 50% yield loss but not this year. With abundant production, we are taking our produce to large farmer markets in Birmingham and Decatur, not limiting ourselves just to the farm stand.” (Surveyed on July 5, 2012)
Danny Dickie, vegetable producer (Oneonta, AL): “We use the insecticide recommendations from the SE Vegetable Handbook and consult Mel Wade before we make a treatment decision. Without insecticides, we can potentially lose over 80% of our crops to worms and stink bugs that may result in over $20,000 loss per acre.” (Surveyed on May 30, 2012)
Albert Riddle, vegetable producer (Titus, AL): “We use the integrated pest management recommendations provided by Chip East – our Regional Extension Agent. I can lose over 50% of my tomato crop if I did not follow IPM for insect and disease management. That is about $10,000 loss per acre.” (Surveyed on May 23, 2012)
AND it all comes down to writing good reports…
Questions for Dr. A?Thank you for your patience!