Making the Most of Your Data: Strategies for Evaluating Your Program Greta Colombi, NDTAC; and John...
-
Upload
nathan-dickerson -
Category
Documents
-
view
215 -
download
0
Transcript of Making the Most of Your Data: Strategies for Evaluating Your Program Greta Colombi, NDTAC; and John...
Making the Most of Your Data: Strategies for Evaluating Your Program
Greta Colombi, NDTAC; and John McLaughlin, ED
2
How Do You Use Your Title I, Part D, Data?
Do you: Look at your Consolidated State Performance Reports
(CSPRs) each year?
Incorporate what you know into applications for funding?
Make statements about how these tables do not really reflect what goes on in the classroom?
Attempt to explain them to your stakeholders?
Put results away in your files?
Hope that the CSPR will go away before next year?
Why using data is important
How data can be used
Strategies for using data
Activities
3
Agenda
Consider ways you can better use your data
Consider what technical assistance (TA) and support you could provide to your subgrantees to encourage data use
4
Outcomes
5
Data Tells You and Your Programs . . .
Where you’ve been
Where you are
Where you’re going
How to get there
6
Barriers to Using Data
Your program’s data are handled separately from your program.
Your program’s culture does not focus on data.
Gathering data is perceived to be a waste of time.
Staff lack adequate orientation and training in the value of data collection.
Staff have had negative experiences with data collection.
Staff are not aware of other programs’ successes in using data.
Staff think that data are collected “just for the State or the Feds.”
7
Working With What You Have
The same data you collect and report…
Demographics of students (race/ethnicity, age, and gender) Academic performance in reading and mathematics
Academic and vocational outcomes Student and facility counts
Program spending*
can be used for… Accountability Program promotion/marketing Program management and improvement
* States do not report program spending within the CSPR, but should have this information at hand.
8
Functions of Data
Help us identify whether goals are being met (accountability)
Tell our departments, delegates, and communities about the value of our programs and the return on their investments (marketing)
Help us replace hunches and hypotheses with facts concerning the changes that are needed (program management and improvement)
Help us identify root causes of problems (program management and improvement)
9
Program Components by Data Function
Program Accountability
Program Marketing/ Promotion
Program Improvement
Student demographics
Are the appropriate students being
served?
How are you addressing the
needs of diverse learners?
Which students need to be better
served?
Student achievement
Are students learning?
What are students learning? What gains have they
made?
How can we help improve student achievement?
Student academic outcomes
Are students continuing their
education?
What are students doing to continue their education?
How can we help improve student
academic outcomes?
10
Strategies for Improving Data Use
Accountability
– Monitor data based on national benchmarks
– Set State benchmarks and monitor program performance
Program Improvement
– Evaluate program (formative and/or summative)
Marketing
– Develop and distribute State/program report cards
11
Data Use Improvement Activities
Meet with SEA staff (data, programmatic) to analyze the data you have
Request disaggregated data from subgrantees/ programs to improve data use
Communicate findings with subgrantees
Support subgrantee/program evaluations
– Communicate allowability of funding
– Include evaluation requirements in program applications/formal agreements
– Provide TA at conferences/meetings or during monitoring on benefits of using data and how to do so
12
Data Use Model
Two Components of the Model:
Data Analysis
and
Program Improvement
Model developed by the National Reporting System for Adult Education support project at the American Institutes for Research
13
14
Focusing the Question
Break the question into inputs and outcomes: Inputs (what your program contributes):
Teacher education, experience, full-time/part-time
Instructional curriculum
Hours of instruction per week
Outcomes (indicators of results): Improved posttest scores
Completed high school
Earned GED credentials
15
Focusing/Refining the Question (1)
Poor Question:
Does my program have good teachers?
Good Question:
Does student learning differ by teacher?
Better Question:
Do students in classes taught by instructors who have more teaching experience have higher test scores than those taught by new teachers?
16
Focusing/Refining the Question (2)
Poor Question:
Is my program helping the most needy students?
Good Question:
Are students who are below grade level learning less in my program than other students?
Better Question:
Are students who are below grade level advancing levels at the same rate as students at grade level?
17
Developing a Data Analysis Plan
What data do you already have that will answer your question?
What additional data, if any, will you need to answer your question?
If so, where will you get the additional data?
What’s your plan for obtaining the data you need—and what’s your timeline?
18
Analyzing and Interpreting Your Data
Keep your original question in mind.
Look for patterns and differences.
Use appropriate data and statistics.
Disaggregate the data.
Consider data quality.
Draw appropriate conclusion(s).
Remember serendipity: Be open to the unexpected.
19
Presenting Your Data (1)
Frequency Tables
Show numbers and percentages by category, e.g., ethnicity, gender, age.
Provide crosstabulations, e.g., ethnicity by age.
20
Presenting Your Data (2)
Graphs and Charts
Bar Chart: Categories are displayed as bars, e.g., students
by age.
Pie Chart: A slice of the pie shows proportion of the
whole, e.g., various ethnicities of total students.
Line Chart: Data form a continuous measure/trend (not
categories), e.g., posttest scores.
21
Presenting Your Data (3)
Communication Strategies
Article by education reporter in local newspaper
Public meeting or news conference presented by superintendent or dean
Newsletters Special events, e.g., open house Web sites Annual report
22
Conclusion
Using your data can help you (1) ensure accountability, (2) make program improvements, and (3) market your program.
It is key to look at your data, involve others, and consider how you can use the data.
You can integrate data use activities in regularly scheduled activities.
23
Activities
Activity 1
– Discuss possible reasons for the scenarios included in the handout
– Consider:
(1) What additional data would you need to better understand the root cause of the problem? and
(2) What could be done about it?
24
Activities
Activity 2
– Review the scenario
– Consider how you could address the issue(s) in regularly scheduled activities