Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.
-
Upload
cody-graves -
Category
Documents
-
view
212 -
download
0
Transcript of Evaluation Design Designing & Refining Questions Ben Silliman Youth Development Specialist NC 4-H.
Evaluation DesignDesigning & Refining Questions
Ben SillimanYouth Development Specialist
NC 4-H
Who Knows What?
• Youth (over age 3) can report their– Feelings, interests, experiences, goals, opinions/
attitudes on knowledge and skills– (reliable observers may give opinions on these)
• General Rules for who answers what:– You can’t know what you don’t know (thus
retrospective or observer ratings usually better)– No one knows your feelings, experiences, and
attitudes/aspirations better than you(let respondents speak for themselves)
Who Knows What?
• Youth (over age 3) can complete valid, developmentally appropriate tests of– Knowledge, Skill, Practice(reliable observers may also give opinions on
these)
• How to know what others know– Demonstration, description, or defense
(authentic assessment) gives best evidence…esp. if judged by trained, reliable observers using the same criteria given to youth
Thinking About Questions
Adjust for Differences in Self-Administered vs. Interviewer-Administered Questionnaires
• Self: make it simple (Eighth Grade), straightforward, and logical. Q1-> Q2 (complex skip patterns, crowded pages, complex instructions delay and discourage respondents)
• Interviewer: accommodate skip patterns, allow for diverse lines of questioning, interaction about ideas, objects, events (note interviewer skills)
Thinking About Questions
Order questions for smooth flow– Be forthcoming about purpose and
sponsor– Group topic or type of questions– Move from general to specific questions– Ask the easy questions first-build rapport– Ask more complex or thought-provoking
questions near the middle—use rapport– Ask sensitive questions near the end-ditto
Thinking About Questions
Phrase questions carefully to avoid bias
• “Leading” the respondent– What do you like about… vs
– What, if anything, do you like…(not assuming respondents like
something)
Thinking About Questions
Keep audience focused• Avoid multiple-option items
– Did public speaking help you gain confidence and skill?
• Avoid complex phrases(Answer Never-Seldom-Often-Always)– This 4-H program did not have an
appropriate level of discipline
Thinking About Questions
Adjusting questionnaire length, remembering that– A relevant, purposeful study can
motivate cooperation and thought– A clear and smooth-flowing routine can
focus and energize respondents– -A respectful and gracious approach can
affirm respondents, making them feel valued
Which Questions to Use
• Open-ended: no set responseIs there anything else you would like to
tell about your 4-H camping experience?– Communicates listening– Allows evaluator to see how
respondents prioritize and organize information
Which Questions to Use
• Close-ended: set response choicesFor the 4-H project areas listed, rank (1st,
2nd, 3rd) each according to your interest:___ Arts & Communication___ Citizenship & Leadership___ Family & Consumer Science___ Health & Nutrition___ Science & Technology
Styles of Close-ended Qs
Likert (esp. for feelings, attitudes)How important are cash prizes in your
decision to participate in 4-H events?Not at all VeryImportant
Important1 2 3 4
5
Styles of Close-ended Qs
Multiple Choice (best or all relevant options)
What is the best thing to do when you plan not to use a gun for a while? (choose one)
a) Cover it with a clothb) Place it on top of a cabinetc) Remove ammunitiond) Call law enforcement
Styles of Close-ended Qs
Ordinal (ranking of options)Rank these topics in order of importance
(1st, 2nd, 3rd, etc) for volunteer training___ Managing a 4-H club___ Teaching using experiential method___ Planning service activities___ Conducting parties___ Mastering program evaluation
Styles of Close-ended Qs
Categorical (identifying by group)Tell us about yourself. I am a a) Girl b) Boy
Numerical (identifying by number)Tell us about yourself. My age is ___ years