Evaluation and Measurement Spring 2010 Quality Training Certificate Program.
-
Upload
tracey-adams -
Category
Documents
-
view
216 -
download
0
Transcript of Evaluation and Measurement Spring 2010 Quality Training Certificate Program.
![Page 1: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/1.jpg)
Evaluation and Measurement
Spring 2010Quality Training Certificate Program
![Page 2: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/2.jpg)
Quality Training Certificate ProgramCertificate Details:
Participants have the option to attend individual classes, or to participate in 6 classes and obtain a certificate in quality training. Classes will be held twice yearly for a total of three years to complete the certificate. Classes will continue to cycle, so new participants will be able to complete the certificate no matter when they begin. Classes can also be taken at any time as stand-alone experiences. There will be no charge for attending the classes, but supervisor permission must be given for attendance during regular work hours.
Attendees from previously held classes will be given credit toward their certificate for attending.
Classes will last 3 hours each, for a total of 18 total contact hours.
Attendance credit will continue to be given for individual classes taken. When attendees complete six classes, they will fill out an application for certification, and send it to BPA, who will verify the information and award the certificate.
![Page 3: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/3.jpg)
Today’s Power Point and HandoutsWill be available online:http://www3.uwstout.edu/bpa/training.cfm
![Page 4: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/4.jpg)
Overview Introductions Goals for Today Opportunity to suggest additions to today’s
agenda Course Content, including:
Practice Designing an Evaluation Discussion Questions Q&A
Breaks as needed
![Page 5: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/5.jpg)
Introductions Tell us:
Who you are What worries you most about evaluation and
measurement?
![Page 6: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/6.jpg)
Goals for Today To understand when evaluation can be used To understand the basic questions that guide
the evaluation process To identify appropriate evaluation
methodologies To implement basic evaluation methodologies To utilize evaluation results
![Page 7: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/7.jpg)
Usage of Today’s Curriculum To apply the skills that you learn today in your
everyday practice Within the next month, to practice at least
one thing that you learned today
![Page 8: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/8.jpg)
Questions/Topics you Want Covered Write down anything you want to be sure we
cover on the notecards in front of you I will collect these when we take our first
break
![Page 9: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/9.jpg)
To understand when eval. can be used
What can be evaluated: 3 P’s:
Programs Processes Products
Evaluations can be “formal” or “informal” and time consuming or simple.
All faculty and staff typically engage in some sort of evaluation in their everyday work – although not all apply it in a formal way.
![Page 10: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/10.jpg)
To understand when eval. can be used
Two major purposes: To answer questions about changes that need
to be made to programs/processes/products (formative),
To answer questions about continuation/discontinuation of programs/processes/products (summative)
Most of the evaluations we do at UW-Stout are formative.
![Page 11: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/11.jpg)
To understand when eval. can be usedExamples: Most grants require an evaluation section The surveys that you fill out at the end of a
training session are a form of evaluation UW-Stout has a formal evaluation plan for our
e-Scholar program, first year experience program, customized instruction program, etc..
Evaluation can be used to determine if you are meeting customer needs
![Page 12: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/12.jpg)
To understand when eval. can be usedDiscussion question: What other examples do you have of when
evaluation can or has been used? What was your experience with these
evaluations?
![Page 13: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/13.jpg)
To understand the basic questions that guide the evaluation process
Two basic questions:• What is the purpose of the process/product/
program?• How will you use the results of the evaluation?
![Page 14: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/14.jpg)
To understand the basic questions that guide the evaluation process
Considerations:• It is sometimes hard to identify the goals and usage
of the results, but you cannot develop a good evaluation without this information. Other ways to ask the question:
• What does success look like? • If this program/process/product did was wildly successful,
describe what that would look like.• Do not proceed with the evaluation if you cannot
articulate how you will use the results.
![Page 15: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/15.jpg)
To understand the basic questions that guide the evaluation process
Discussion Question: How often have you been asked to participate
in a survey, focus group, or other data collection method when it was not clear how the results would be used?
![Page 16: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/16.jpg)
To understand the basic questions that guide the evaluation process
Goal: How to know if goal is achieved:
Evaluation methods:
Target: Target date:
Goal 1
Goal 2
Goal 3
Goal 4
• All evaluation methods, data, and analysis should tie back to the goals and how the data will be used• A matrix can facilitate this:
![Page 17: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/17.jpg)
Example evaluation plan
•
![Page 18: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/18.jpg)
To understand the basic questions that guide the evaluation process
Practice: Think of an evaluation that you have done, or
that you would like to do, or that you anticipate you will be involved with.
If you don’t have one, use one of the examples on the flip chart.
Fill in: Evaluation question Goals of the process/product or program How the results of the evaluation will be used
![Page 19: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/19.jpg)
To understand the basic questions that guide the evaluation process
Once you have identified your goals and how the results can be used, you need to think about how you will know if the goal is achieved.
At this point, do not think about methods.
Example: Goal: To identify appropriate evaluation
methodologies How achieved: Level of knowledge
participants have of evaluation methodologies
![Page 20: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/20.jpg)
To understand the basic questions that guide the evaluation process
Discussion question: Who has a goal from their sheet that they are
comfortable sharing? How will we know if this goal is achieved?
![Page 21: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/21.jpg)
To understand the basic questions that guide the evaluation process
Practice: Fill in the “how to know if goal is achieved”
column on your sheet
![Page 22: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/22.jpg)
To identify appropriate evaluation methodologies
Considerations: Don’t let methods constrain initial discussion –
start with ideas. People like to challenge methodologies if they
don’t like the results. Too much data is a common problem. All data
must tie back to goals. It often takes more time to collect, manage and
analyze data than you think it will. Utilize multiple methods. Available time and resources will often be the
driving factors for selecting methods. Also your audience.
![Page 23: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/23.jpg)
To identify appropriate evaluation methodologies
Major evaluation methods: Existing data Surveys Focus Groups/Interviews Inventories/Usage data/Database creation
![Page 24: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/24.jpg)
To identify appropriate evaluation methodologies
Discussion questions: What do you think we see most often at
Stout? What do you think is the most underutilized
method at Stout?
![Page 25: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/25.jpg)
To identify appropriate evaluation methodologies
Existing dataYou want to use existing data wherever
possible. Pros:
Minimal time commitment Readily available Ensures data collected is used for multiple
purposes Cons:
May be limited on what is available
![Page 26: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/26.jpg)
To identify appropriate evaluation methodologies
Surveys Use surveys when:
The information you need to obtain is not available from existing data, and cannot easily be obtained through inventories or databases
You don’t need detailed qualitative data You have enough time to do it right
![Page 27: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/27.jpg)
To identify appropriate evaluation methodologies
Surveys Pros:
Can get at factors not easily obtainable from other sources – for example, motivation, attitudes
Typically does not require a large time commitment from participants
Best for likert-scale, checklist short answer-type questions
Cons: Relies on self-report Not good if you want in-depth, qualitative comments Language, response rates, etc… often challenged
![Page 28: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/28.jpg)
Purpose of a focus group:
“The purpose of a focus group is to listen and gather information. It is a way to better understand how people feel or think about an issue, product, or service. Participants are selected because they have certain characteristics in common that relate to the topic of the focus group”
Krueger, Richard, and Casey. Focus Groups. 3rd ed. Thousand Oaks, USA: Sage Publications, 2000.
![Page 29: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/29.jpg)
What a focus group is A carefully planned series of discussions designed
to obtain perceptions on a defined area of interest Discussions in a permissive, nonthreatening,
environment Groups of 5-12 facilitated by a skilled facilitator Repeated groups – need to hold at least 3-4
sessions
![Page 30: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/30.jpg)
When to use focus groups:
Looking for a range of ideas or perspectives on an issue
Trying to determine differences in perspectives between groups of people “A group possesses the capacity to become
more than the sum of its parts” (Krueger and Casey, 2000)
Researcher needs information to aid existing quantitative data
![Page 31: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/31.jpg)
When NOT to use focus groups:
Want people to come to a consensus Want to educate people Don’t intend to use results Other methodologies can produce better
results Other methodologies can produce same
results more efficiently Don’t have enough time to follow standard
focus group procedures
![Page 32: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/32.jpg)
To identify appropriate evaluation methodologies
Focus Group/Interviews Pros
Allows you to go in-depth in an issue Allows for a range of ideas – and you hear from
people in their own voice Cons
Most time-consuming approach
![Page 33: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/33.jpg)
To identify appropriate evaluation methodologies
Inventories/Usage data/Observations Pros:
Typically more accurate than self-report data Good for tracking participation in events,
trainings, programs, etc.. Cons:
Only works for things you can count or observe Often have to rely on other people to obtain the
data When multiple people are involved, need to make
sure that everything is counted in the same way
![Page 34: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/34.jpg)
To identify appropriate evaluation methodologiesPractice: Circle the evaluation methods that you feel
can be utilized to obtain the information in the second column of your sheet. Circle multiple methods if you can use more than one method.
![Page 35: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/35.jpg)
Tying it all together You have two more columns on your sheet – a
target and target date. It is important that you identify specific
targets and dates before you collect any data – as after you collect the data, people will use it to prove their point.
Often you cannot identify a target until you have your evaluation methods more fully developed – so we will not be filling in these columns today.
![Page 36: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/36.jpg)
Tying it all together What you have on your sheets now is typically
your ideal scenario for how you can go about evaluating your process/product/program.
Usually, you don’t have time to do it all, so next you need to determine what is feasible given your time, resources, and who will be reviewing the results.
![Page 37: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/37.jpg)
Tying it all togetherStart by sharing your plan with your key
stakeholders to get their input. Explain why you have chosen the methods that
you have. Convince them that you have good reason for choosing the methods that you did.
What concerns do they have about the plan? Given your available time and resources, you need
their help to prioritize what is most important to them.
It is often helpful to obtain approval of the plan in writing.
![Page 38: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/38.jpg)
Tying it all togetherPractice: Share your plan with a neighbor. Pretend
they are one of your key stakeholders. Get their input.
![Page 39: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/39.jpg)
Tying it all togetherDiscussion question: Is anyone willing to share their plan with the
group?
![Page 40: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/40.jpg)
To implement basic evaluation methodologies
Now that you have approval on your plan, you need to implement it.
Next, you need to develop your instruments and implement them.
![Page 41: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/41.jpg)
To implement basic evaluation methodologies
Considerations: Before collecting your data, develop a data
analysis plan. Just as it is important to get approval on your
evaluation plan before starting, it is also important to obtain approval on the instrument and data analysis plan.
Pilot testing is important – administer the survey/focus group/form to people similar to those who will receive it.
![Page 42: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/42.jpg)
To implement basic evaluation methodologies
Existing Data Can sometimes find a survey, focus group or
other study that has recently been administered that covers the same topics that you are assessing
Several ways to access existing data at Stout: Contact the BPA office Survey Clearinghouse (in development) Information Portal (in development) Qualitative results:
http://www.uwstout.edu/static/bpa/ir/surveylistqual.html
![Page 43: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/43.jpg)
To implement basic evaluation methodologies
Surveys Qualtrics is the online survey instrument that
Stout uses: http://www3.uwstout.edu/bpa/survey/
Qualtrics question library in development. Will be available from the above link. Will provide suggested demographic questions Will also provide suggestions for other standard
questions Typically don’t want it to take more than 10
minutes to complete – time to complete is more important than the number of questions.
![Page 44: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/44.jpg)
To implement basic evaluation methodologies
Surveys, cont. Look for other surveys you can model after –
internally developed and externally developed. Request permission from the author to use some of the same questions. Survey Clearinghouse (in development) Surveys webpage:
http://www.uwstout.edu/static/bpa/ir/surveylist.html
Sample survey guide: http://www.uwstout.edu/static/bpa/ir/afu/information/instruction.pdf
![Page 45: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/45.jpg)
To implement basic evaluation methodologies
Focus Groups/Interviews Different than a “listening session”
Discussion question:How are focus groups different than listening
sessions?
![Page 46: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/46.jpg)
To implement basic evaluation methodologies
Core Ingredients Continue groups until you hear same themes
repeated – the ARC typically holds 8 sessions. Sometimes it is possible to do less.
No more than 12 people in a group Circle seating Typically requires formal training for
facilitators/ moderators Pre-determined questions Systematic analysis
![Page 47: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/47.jpg)
To implement basic evaluation methodologies
Inventories/Usage data/Observations Must provide training to the people who will
be entering data into databases or inventories. Include specific instructions.
Examples: Sign-in sheets outside of a tutor center (sample in
separate document) Appointment logs Formal observation forms, with checklists, open-
ended questions, tally marks, etc… (sample in separate document)
![Page 48: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/48.jpg)
To utilize evaluation results
Evaluation results are primarily used for two purposes:
Program/process/product improvement (formative)
Decisions about program/process/product continuation/discontinuation (summative)
Both are important
![Page 49: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/49.jpg)
To utilize evaluation results
Formative evaluation: Typically done early in on program. Usually
identifies improvements that can be made related to program implementation.
![Page 50: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/50.jpg)
To utilize evaluation results
Formative evaluation, examples: Decisions about improvements that can be
made to get more people to attend a pre-college program.
Decisions about improvements that can be made to get more people to apply what they’ve learned from training programs.
Decisions about new ways to present or deliver departmental newsletters.
![Page 51: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/51.jpg)
To utilize evaluation results
Summative evaluation: Typically done later on in the program – after
all of your target dates have passed. Usually used to make decisions about continuation/discontinuation.
Typically summative decisions are the most difficult to make. If you establish specific targets and target due dates, and have done formative evaluation prior to the summative evaluation, these decisions are easier.
![Page 52: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/52.jpg)
To utilize evaluation results
Summative Evaluation, examples: Decisions about if funding for a grant will
continue Decisions about if various initiatives to
improve retention will continue Decisions about if on campus funding will
become permanent Decisions to keep or eliminate items from the
campus-wide training and development plan
![Page 53: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/53.jpg)
To utilize evaluation results We often hear from people that nothing is
done with the feedback they provide on surveys, focus groups, etc…
It is important to invest time into sharing how you have used the results from the evaluation. People will not often make the connection between feedback and their actions on their own.
![Page 54: Evaluation and Measurement Spring 2010 Quality Training Certificate Program.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649dc55503460f94ab885d/html5/thumbnails/54.jpg)
Questions/Comments?