PLT Program Evaluation
description
Transcript of PLT Program Evaluation
PLT PROGRAM EVALUATION
What do funders want to know and can you (provide the data to) give
it to them?
Norie Dimeo-EdigerCarla Rapp
Michelle Youngquist
HERE’S WHAT OUR STAKEHOLDERS SAID
We would like to know that your forest education programs for educators (PLT workshops, tours, etc.) are effective and
a good investment of our money.$$$$$$$$$$$$
HERE IS WHAT WE HEARD
Conduct an extensive (and expensive) statistically
valid evaluation.
HERE IS WHAT WE THOUGHT
I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant.
Robert McCloskey
What did the stakeholders mean?
?????????
So………
Norie asked Pete and Paul.
Carla asked Larry, Winston and Bill.
Michelle asked Kevin, Bob and Mike.
What did they say????
I want to know that teachers and students “get it.”
-It would be nice to be able to track from grade to grade and year to year the attitudes and understanding of the students regarding our issues.
-I want you to go out and tape the teachers using the activities so we can see them in action.
Are they using the materials?
Has it changed (for the better) the way they teach about forests?
I want to know enough people have responded to the questionnaire/survey that I can trust the results
Is PLT still the best way to deliver our message and how do we evaluate that?
How do we do what they want?
LEVELS OF EVALUATION Level 1. Reaction – What is the participants’
response to the project or activity?
Level 2. Learning – What did the participants learn?
Level 3. Behavior or Application – Did the participants’ learning affect their behavior?
Level 4. Results – Did participants’ behavior change move the original situation towards the objective
Source: Kirkpatrick (1994). Designing Evaluation for Education Projects – NOAA
LEVEL 1. REACTION – WHAT IS THE PARTICIPANTS’ RESPONSE TO THE PROJECT OR ACTIVITY?
LEVEL 2. LEARNING – WHAT DID THE PARTICIPANTS LEARN?
When asked to rate their content knowledge before/after the workshop the responses were:
BEFORE High 5 4 3 2 1 Low 1 5 8 10 5 1 AFTER High 5 4 3 2 1 Low 17 13 0 0 0
LEVEL 3. BEHAVIOR OR APPLICATION – DID THE PARTICIPANTS’ LEARNING AFFECT THEIR BEHAVIOR?
LEVEL 4. RESULTS – DID PARTICIPANTS’ BEHAVIOR CHANGE MOVE THE ORIGINAL SITUATION TOWARDS THE OBJECTIVE
Oil City is a Title I school in rural northwest Louisiana. In 2001, faced with declining enrollment and a possible closure, teachers and the administration worked together to create a school with an environmental focus.
Measurable AchievementThe success of the new environmental education focus and the
activities that sprang from it brought recognition to the school. In 2004–2005, the school was recognized by the Louisiana Association of Business and Industry as one of the “Top 10” most improved schools in Louisiana.
In 2006, the school received the National School Change Award sponsored by Fordham University Graduate School of Education, the American Association of School Administrators, and Pearson Education. Students have been invited to perform at the Louisiana Forestry Association Conference in Shreveport.
In 2006, a fourth-grade teacher at the school, Brenda Smith, was named a National Outstanding Educator for Project Learning Tree.
SOME EVALUATION METHODS THAT MIGHT ANSWER THEIR QUESTIONS
Focus groups Questionnaires or surveys Observation Case studies
FOCUS GROUPS Purpose-to explore a topic in depth
through group discussion Advantages-key info can be conveyed,
efficient way to get range and depth of info
Disadvantages-can be hard to analyze, need good facilitator, can be hard to schedule
QUESTIONNAIRES AND SURVEYS Purpose-to quickly/easily obtain a lot of
info from people in a non-threatening way.
Advantages-anonymous, inexpensive, large sample size, easy to compare and analyze
Disadvantages-might not get careful feedback, wording can bias response, may not tell full story, low return rate
OBSERVATION Purpose-to gather accurate info on how
people behave Advantages-generates data about
behavior not reported behavior, data collected in context
Disadvantages-time intensive, “observer effect”, small sample size
CASE STUDIES Purpose-to conduct comprehensive
examination through cross comparison of cases
Advantages-provides vivid imagery, and detailed understanding of the experience
Disadvantages-time intensive, detailed understanding of a particular participant results can not be generalized.
COMMUNICATION OF RESULTS What? To whom? How much? How
often? What language? eg. Edu-speak?
Communicating results should not be an after thought, but part of the evaluation plan.
SOME EXAMPLES OF WHAT WE’VE DONE Carla: Pre/Post “Viewpoints on the
Line” adaptation, video-taping teachers doing PLT
Norie: Professional 3rd party evaluation Michelle: Surveys, portfolio of teacher
& student work. Collecting, but:
You do need to communicate it to the stakeholders. How best to do that?
WHAT HAVE OTHERS EXPERIENCED
QUESTIONS?
Thank you! Norie Dimeo-Ediger, Oregon Forest Resources Institute
[email protected] 971-673-2956 www.OregonForests.org
Carla L. Rapp, Georgia Forestry Association [email protected] 478-992-8110 www.gfagrow.org
Michelle Youngquist, Idaho Forest Products Commission [email protected] 208-334-4061 www.idahoforests.org
CARLA, MICHELLE & NORIE