Intro to NRS Data DivingIntro to NRS Data DivingMary A. Gaston, Ed.D. &Jennifer Cooper-Keels

February 4, 2011

22

Why Look at Data?

Data help us to…

• Replace hunches and anecdotes with facts concerning the changes that are needed;

• Identify root causes of problems;

• Identify whether student or program goals are being met; and

• Tell our stakeholders, including students, about the value of our programs and the return on their investments.

04/18/23

33

Data: A Carrot or a Stick?

Data may be used…

• To highlight, clarify, and explain what’s happening in your program

OR

• To show what’s not happening in your program.

“However beautiful the strategy, you should occasionally look at the results.”

–W. Churchill

04/18/23

44

Data Tell You

• Where you’ve been• Where you are• Where you’re going• How to get there

Data can help you design a quality program to help meet learners’ goals.

04/18/23

55

Applied to Adult Education…

What can data do?• Guide you to improve instruction

• Measure program success & effectiveness

• Tell you if what you are doing is making a difference

• Tell you which classes are getting the results you want—and which are not

• Get to the root of problems, such as poor retention, low educational gains, or low transition rates

04/18/23

Starting the DiveStarting the Dive

77

Starting the Dive

• Attendance

• Educational Gain

• Transition Outcomes (Goals)

04/18/23

For this workshop, we will focus on “Attendance” and “Educational Gain.”

3 Main Measures in our Data System:

88

Attendance

• Contact hours of instruction the learner receives (NRS)

• Includes intensity and duration• Can help to tell us whether:

– Instruction is successful– Content and materials are relevant– Students are motivated– Students are reaching their goals

04/18/23

99

Examples: What Increases Attendance

• Quality instruction and relevant content

• Well-trained teachers

• Clear goals set at intake, revisited regularly, and matched to teachers and content

• Reduction of obstacles – flexibility in programming, support services, and access to site off-hours

(NCREL; Lieb, 1991; Comings, 2007; Beder, 1988; Beder, 1991; Comings, Parella, & Soricone, 1999; Kerka, 2005; Thoms, 2001; Porter, Cuban & Comings, 2005)

04/18/23

1010

Educational Gain

• Advancement through 12 educational functioning levels

• Core NRS measure • Can tell us:

– Whether the program/students are meeting goals – Which sites/classes/teachers are most effective – Extent of student progress– Impact of changes

04/18/23

1111

Examples: What Increases Ed Gain

• Make classes learner-centered

• Focus on relevant knowledge

• Opportunity for practice and application

• Coherence

• Sufficient Intensity and Duration 

(NRC, 1999; Garet, Porter, Desimone, Birman, & Yoon, 2001)

04/18/23

1212

Do You Trust Your Data?

Data analysis is only as good as the original data allow.

Keys to good data collection systems include:• Clear policies and procedures for data entry• Data is entered & reviewed daily, weekly, or monthly• Teachers, staff, administrator all have access to data

and review regularly• Teachers share data with students

What does your program do to ensure data is accurate, reporting is timely, and staff have access to the data?

What does your program do to ensure data is accurate, reporting is timely, and staff have access to the data?

04/18/23

Take a Dip in the Data PoolTake a Dip in the Data Pool

1414

Dive into the Data Pool

For each of the next few slides write down your observations for discussion

• What do you see?

• What is interesting or unusual?

• Do any questions or hypotheses come to mind as a result?

04/18/23

1515

Write Observations/Questions?

04/18/23

100

1616

Write Observations/Questions?

04/18/23

ABE

ASE

ESL

0%

10%

20%

30%

40%

50%

60%

70%

80%

Local Program State

Average

47%

40%

75%

48%

63%

43%

Percentage of Students Who Completed One or More Levels 2008-09

ABE

ASE

ESL

0%

10%

20%

30%

40%

50%

60%

70%

80%

Local Program State

Average

52% 53%

71%

52%

50%

41%

Percentage of Students Who Completed One or More Levels 2007-08ABE

ASE

ESL

0%

10%

20%

30%

40%

50%

60%

70%

Local Program State

Average

37%

28%

66%

43%

54%

31%

Percentage of Students Who Completed One or More Levels 2009-10

1717

Write Observations/Questions?

04/18/23

1818

Write Observations/Questions?

04/18/23

1919

Write Observations/Questions?

04/18/23

2020

Write Observations/Questions?

04/18/23

Satellite R Main PM

Young Adult

Main AM

2121

Write Observations/Questions?

04/18/23

2222

Where to Go From Here?

04/18/23

•What should I change or replicate?

•What data supports this change? What additional data should be reviewed?

•What is the timeframe for change? Is it realistic?

•What obstacles/barriers will we encounter?

•What is the follow-up plan to measure and evaluate change?

Based on what was learned from this “data dive”:

2323

You Just Did a Data Dive!

04/18/23

How well is our program helping students move to the next level?

Look at the ranges of completion rates by class in

each level (Gain 16).

Compare each completion level to peerprograms and the state

averages (Gain 9).

For narrow ranges

For wide ranges

Are these numbers consistent with past years' performance?

(Gain 17)

Yes

No

Which classes have the lowest pre/post test percentages?(Gain 1)

Which have the lowest rates of educational gain? (Gain 8)

Which have the lowest attendance/retention rates? (Attendance 6)

Are these the same classes? (Gain 20)

Is there anything program-wide (e.g., an attendance policy) that has changed in the last

year?

No

Do disaggregations

over time.

For low relative performance

No.

Yes

Begin with the classes with the lowest gain.

See if you can identify plausible reasons for the

low gain. Identify whether there are any outliers on the

bottom end and consider actions, such as

providing technical assistance, that might

improve performance.

Is there a policy, practice, teacher, subgroup , etc. that seems to be driving those numbers? Consider actions, such as providing technical assistance, that might improve performance.

ALSO--Identify the best performers and see what they have in common and whether those things may be replicated or built upon.

Yes

Assess whether you want to keep this change in place.

For high relative performance

Are the data of high quality?(e.g., see Gain #1-6).

Stop and consider doing disaggregations to find the sources of error.No

Yes

2424

Questions: AttendanceAttendance & Retention

Sample questions Further questions

Data collection & quality

Who enters attendance data at each site? How often is attendance data entered?

Who checks the data? How often?

Students How does attendance differ by student type (ESL vs. ABE)?

When in the term do students tend to drop/stop-out most? Is this the same across sites?

Teachers Which classes have very high (or low) attendance?

Do teachers with high attendance have greater educational gains?

Instruction Does attendance vary by instructional content (e.g. GED, workplace) or level?

How many hours does it take to achieve a goal, on average?

Program What is the average attendance for my program?

Are my program’s attendance hours similar to other programs?

Program policy Are my managed enrollment classes more successful than open classes?

Does managed enrollment result in higher ed gains or greater goal achievement?

2525

Questions: Educational GainEducational Gain Sample question Further questions

Data collection & quality

What is the range of pre/posttest scores in my program/site?

Are all the test scores within the correct range for the test and class level?

Students Which students are most likely to complete a level (student characteristics)?

Do students with higher contact hours have greater completion rates?

Teachers What teacher characteristics are most related to level completion?

How high is teacher turnover at each site? Which sites retain teachers longest/best?

Instruction Which instructional approaches have the greatest impact on gain?

Do assessments match course content?

Program How many hours of PD do our teachers participate in?

Which PD have the greatest impact on student learning?

Program policy Do placement policies differ among sites?

Which placement policies have an impact on educational gains?

2626

Tools for Data Diving

04/18/23

2727

What Do I Want to Know?

What questions do you want to answer about your own local program?

04/18/23