PD 360 Impact Assessment:Updated Findings

regarding the Impact of PD 360 on

Student Proficiency Rates

Prepared by Steven H. Shaha, PhD, DBA

1

PD 360 Impact AssessmentPD 360 Impact AssessmentExecutive Summary: Initial FindingsExecutive Summary: Initial Findings

• Statistically significant* advantages were verified favoring schools with PD 360 versus District Benchmarks.– 4th Grade and 8th Grade for:

• Math proficient and advanced (p<.001)

• Reading proficient and advanced (p<.001)

2* Statistical significance establishes genuine differences between groups and verifies that impacts were “real” and not merely due to chance and, in this case, due to any pre-existing biases in group differences. The appropriate p-values are included with all differences explained herein.

Impacts of PD 360 for4th Grade

3

Note to Reader: To better dramatize the magnitude of the consistently favorable impact of PD 360, graphics included hereafter represent a variety of perspectives and a sampling of different interpretive insights, and not an exhaustive nor uniformly arrayed set of results.

53.554.054.555.055.556.056.557.057.558.0

Prior Year District Benchmark

Post-PD 360 Year

Math Proficient - 4th Grade

High Users

Low Users

Improvements in Math Proficiency for PD 360

4

High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)

and Low Utilizing School and Low Utilizing School improved by 2.0%improved by 2.0% (p<.01)

High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)

and Low Utilizing School and Low Utilizing School improved by 2.0%improved by 2.0% (p<.01)

33.033.534.034.535.035.536.036.537.037.538.038.5

Prior Year District Benchmark

Post-PD 360 Year

Math Advanced - 4th Grade

High Users

District Avg

Improvements in Math Advanced for PD 360

High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 9.0% performance gap of 9.0% (p<.001)

High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 9.0% performance gap of 9.0% (p<.001)

5

High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 1.4% by 1.4% (p<.05)

While the district fell by 7.0% While the district fell by 7.0% (p<.001)

High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 1.4% by 1.4% (p<.05)

While the district fell by 7.0% While the district fell by 7.0% (p<.001)

60.0

62.0

64.0

66.0

68.0

70.0

72.0

Prior Year District Benchmark

Post-PD 360 Year

Reading Proficient - 4th Grade

High Users

Low Users

Improvements in Reading Proficiency for PD 360

High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)

High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)

6

Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)

Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)

High Utilizing Schools significantly High Utilizing Schools significantly outperformed the Low Utilizing outperformed the Low Utilizing Schools by 2.9% Schools by 2.9% (p<.01)

High Utilizing Schools significantly High Utilizing Schools significantly outperformed the Low Utilizing outperformed the Low Utilizing Schools by 2.9% Schools by 2.9% (p<.01)

56.0

58.0

60.0

62.0

64.0

66.0

68.0

70.0

72.0

Prior Year District Benchmark

Post-PD 360 Year

Reading Proficient - 4th Grade

High Users

Low Users

District Avg

Improvements in Reading Proficiency for PD 360

7

High Utilizing Schools created a High Utilizing Schools created a performance gap of 14.0% performance gap of 14.0% (p<.001)

High Utilizing Schools created a High Utilizing Schools created a performance gap of 14.0% performance gap of 14.0% (p<.001)

High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)

While the district fell by 3.2% While the district fell by 3.2% (p<.001)

High Utilizing Schools significantly High Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 10.3% district benchmark by 10.3% (p<.001)

While the district fell by 3.2% While the district fell by 3.2% (p<.001)

Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)

Low Utilizing Schools significantly Low Utilizing Schools significantly outperformed the collective outperformed the collective district benchmark by 7.2% district benchmark by 7.2% (p<.001)

Impacts of PD 360 for8th Grade

8

Note to Reader: To better dramatize the magnitude of the consistently favorable impact of PD 360, graphics included hereafter represent a variety of perspectives and a sampling of different interpretive insights, and not an exhaustive nor uniformly arrayed set of results.

50.0

51.0

52.0

53.0

54.0

55.0

56.0

Prior Year District Benchmark

Post-PD 360 Year

Math Proficient - 8th Grade

High Users

Low Users

Improvements in Math Proficiency for PD 360

9

High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)

and Low Utilizing School and Low Utilizing School improved by 2.6%improved by 2.6% (p<.01)

High Utilizing Schools improved High Utilizing Schools improved by 4.9% over the District by 4.9% over the District benchmark benchmark (p<.001)

and Low Utilizing School and Low Utilizing School improved by 2.6%improved by 2.6% (p<.01)

20.021.022.023.024.025.026.027.028.029.030.0

Prior Year District Benchmark

Post-PD 360 Year

Math Advanced - 8th Grade

High Users

Low Users

Improvements in Math Advanced for PD 360

10

High Utilizing Schools improved High Utilizing Schools improved by 28.8% over the District by 28.8% over the District benchmark benchmark (p<.001)

and Low Utilizing School and Low Utilizing School improved by 5.6%improved by 5.6% (p<.001)

High Utilizing Schools improved High Utilizing Schools improved by 28.8% over the District by 28.8% over the District benchmark benchmark (p<.001)

and Low Utilizing School and Low Utilizing School improved by 5.6%improved by 5.6% (p<.001)

56.0

58.0

60.0

62.0

64.0

66.0

68.0

70.0

Prior Year District Benchmark

Post-PD 360 Year

Reading Proficient - 8th Grade

PD 360 Users

District Avg

Improvements in Reading Proficiency for PD 360

11

High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 11.9% performance gap of 11.9% (p<.001)

High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 11.9% performance gap of 11.9% (p<.001)

PD 360 Utilizing Schools PD 360 Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 8.3% by 8.3% (p<.001)

While the district fell by 3.2% While the district fell by 3.2% (p<.001)

PD 360 Utilizing Schools PD 360 Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 8.3% by 8.3% (p<.001)

While the district fell by 3.2% While the district fell by 3.2% (p<.001)

30.030.531.031.532.032.533.033.534.034.5

Prior Year District Benchmark

Post-PD 360 Year

Reading Advanced - 8th Grade

High Users

District Avg

Improvements in Reading Advanced for PD 360

12

High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 8.1% performance gap of 8.1% (p<.001)

High Utilizing Schools High Utilizing Schools significantly outperformed significantly outperformed collective district change collective district change between years to create a between years to create a performance gap of 8.1% performance gap of 8.1% (p<.001)

High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 5.7% by 5.7% (p<.001)

While the district fell by 2.2% While the district fell by 2.2% (p<.05)

High Utilizing Schools High Utilizing Schools significantly outperformed the significantly outperformed the collective district benchmark collective district benchmark by 5.7% by 5.7% (p<.001)

While the district fell by 2.2% While the district fell by 2.2% (p<.05)

PD 360 Impact Assessment:Former Findings

regarding the Impact of PD 360 on

Student Proficiency Rates

Prepared by Steven H. Shaha, PhD, DBA

13

PD 360 Impact AssessmentPD 360 Impact AssessmentExecutive Summary: Initial FindingsExecutive Summary: Initial Findings

• Statistically significant and resounding advantages were identified favoring schools with PD 360.

• The higher the quantified utilization, the greater the statistical advantage.– High utilization led to performance advantages ranging

from twicefrom twice to 15 timesto 15 times the gains in proficiency rates versus district averages (p<.01 and p<.001, respectively).

– Low utilization led to gains in proficiency rates from 10% from 10%

to twice to twice the district averages (p<.01 and p<.001, respectively).

14

Improvements in Proficiency Rates Correlated with PD 360

0.00

0.50

1.00

1.50

2.00

2.50

3.00

School Math Change School Reading Change

Comparative Change forLow vs. High Utilization

Low Utilization

High Utilization

Districts Average

Low Utilization

High Utilization

Districts Average

Percent Advantage for High vs.

Low Utilization

Statistical Significance

School Math Change 1.18 2.54 0.16 115.3% p<.001 2.2times greater improvement

School Reading Change 1.22 2.13 1.12 74.4% p<.001 1.7times greater improvement

An advantage of:

High Utilization results in more the TWICE the gains in Math proficiency rates than does Low Utilization (p<.001)

High Utilization results in nearly TWICE the gains in Reading proficiency rates than does Low Utilization (p<.001)

High Utilizers Outperformed all othersHigh Utilizers Outperformed all others

But even low utilizers outperformed no PD 360

15

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Math Change Reading Change

Axi

s Ti

tle

Comparative Change forHigh Utilization vs. Districts Average

High Utilization Schools

Districts Average

High Utilization

SchoolsDistricts Average

Percent Advantage

Statistical Significance

Math Change 2.54 0.16 1484.7% p<.001 15.8times greater improvement

Reading Change 2.13 1.12 90.6% p<.001 1.9times greater improvement

An advantage of:

High Utilization results in more the FIFTEEN times the gains in Math proficiency rates than the Districts average (p<.001)

High Utilization results in nearly TWICE the gains in Reading proficiency rates than the Districts average (p<.001)

High Utilizers High Utilizers outperformed no PD 360

15.8 times greater gains in Math vs. District

90.6% greater gains in Reading vs. District

16

Improvements in Proficiency Rates Correlated with PD 360

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Math Change Reading Change

Comparative Change forLow Utilization vs. Districts Average

Low Utilization Schools

Districts Average

Low Utilization results in more the SEVEN times the gains in Math proficiency rates than the Districts average (p<.001)

Low Utilization results in nearly 10% greater gains in Reading proficiency rates than the Districts average (p<.01)

But even Low Utilizers Low Utilizers outperformed no PD 360

7.4 times greater gains in Math vs. District

9.3% greater gains in Reading vs. District

Low Utilization

SchoolsDistricts Average

Percent Advantage

Statistical Significance

Math Change 1.18 0.16 636.1% p<.001 7.4times greater improvement

Reading Change 1.22 1.12 9.3% p<.01 1.1times greater improvement

An advantage of:

17

Improvements in Proficiency Rates Correlated with PD 360

High Min/User Schools

Districts Average

Percent Advantage

Statistical Significance

Math Change 1.91 0.16 1096.6% p<.001 12.0times greater improvement

Reading Change 1.92 1.12 71.5% p<.001 1.7times greater improvement

An advantage of:

0.00

0.50

1.00

1.50

2.00

2.50

Math Change Reading Change

Comparative Change forHigh Min/User Schools vs. Districts

High Min/User Schools

Districts Average

High Minutes per User result in more the TWELVE times the gains in Math proficiency rates than Districts avg. (p<.001)

High Minutes per User result in nearly TWICE the gains in Reading proficiency rates than the Districts average (p<.001)

More Minutes per User More Minutes per User resulted in Higher Performance

12.0 times greater gains in Math vs. District

71.5% greater gains in Reading vs. District

18

Improvements in Proficiency Rates Correlated with PD 360

0.000.200.400.600.801.001.201.401.601.80

Math Change Reading Change

Comparative Change forLow Min/User Schools vs. Districts

Low Min/User Schools

Districts Average

Low Min/User Schools

Districts Average

Percent Advantage

Statistical Significance

Math Change 1.67 0.16 943.4% p<.001 10.4times greater improvement

Reading Change 1.49 1.12 32.8% p<.001 1.3times greater improvement

An advantage of:

Low Minutes per User result in more the TEN times the gains in Math proficiency rates than the Districts average (p<.001)

Low Minutes per User result in more than 30% greater gains in Reading proficiency rates than the Districts average (p<.001)

But even Low Minutes per User Low Minutes per User outperformed no PD 360

10.4 times greater gains in Math vs. District

32.8% greater gains in Reading vs. District

19

Improvements in Proficiency Rates Correlated with PD 360

High Pct of Users that

Viewed Schools

Districts Average

Percent Advantage

Statistical Significance

Math Change 2.17 0.16 1254.2% p<.001 13.5times greater improvement

Reading Change 2.19 1.12 95.8% p<.001 2.0times greater improvement

An advantage of:

0.00

0.50

1.00

1.50

2.00

2.50

Math Change Reading Change

Comparative Change forHigh Pct Viewer Schools vs. Districts

High Pct of Users that Viewed Schools

Districts Average

High Percentage of Users that Viewed resulted in more the THIRTEEN times the gains in Math proficiency rates than the Districts average (p<.001)

High Pct of Users that Viewed resulted in TWICE the gains in Reading proficiency rates than the Districts average (p<.001)

Higher percentage of Viewers Higher percentage of Viewers resulted in Higher Performance

13.5 times greater gains in Math vs. District

95.8% greater gains in Reading vs. District

20

Improvements in Proficiency Rates Correlated with PD 360

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

Math Change Reading Change

Comparative Change forLow Pct Viewer Schools vs. Districts

Low Pct of Users that Viewed Schools

Districts Average

Low Pct of Users that

Viewed Schools

Districts Average

Percent Advantage

Statistical Significance

Math Change 1.50 0.16 836.5% p<.001 9.4times greater improvement

Reading Change 1.38 1.12 23.6% p<.001 1.2times greater improvement

An advantage of:

Low Percent of Users that Viewed resulted in more the NINE times the gains in Math proficiency rates than the Districts average (p<.001)

Low Percent of Users that Viewed resulted in more than 20% greater gains in Reading proficiency rates than the Districts average (p<.001)

But even Low percentage of Viewers Low percentage of Viewers outperformed no PD 360

9.4 times greater gains in Math vs. District

23.6% greater gains in Reading vs. District

21

Improvements in Proficiency Rates Correlated with PD 360

0.00

0.50

1.00

1.50

2.00

2.50

Math Change Reading Change

Comparative Change forSchools with High # of Viewers vs.

Districts

High Pct of Users that Viewed Schools

Districts Average

High Number of Users that

Viewed Schools

Districts Average

Percent Advantage

Statistical Significance

Math Change 2.14 0.16 1237.1% p<.001 13.4times greater improvement

Reading Change 2.16 1.12 92.5% p<.001 1.9times greater improvement

An advantage of:

High Number of Users that Viewed resulted in more the THIRTEEN times the gains in Math proficiency rates than the Districts average (p<.001)

High Number of Users that Viewed resulted in nearly TWICE the gains in Reading proficiency rates than the Districts average (p<.001)

More Viewers More Viewers resulted in Higher Performance

13.4 times greater gains in Math vs. District

92.5% greater gains in Reading vs. District

22

Improvements in Proficiency Rates Correlated with PD 360

0.000.200.400.600.801.001.201.401.601.80

Math Change Reading Change

Comparative Change forSchools with Low # of Viewers vs.

Districts

Low Number of Users that Viewed Schools

Districts Average

Low Number of Users that

Viewed Schools

Districts Average

Percent Advantage

Statistical Significance

Math Change 1.55 0.16 870.0% p<.001 9.7times greater improvement

Reading Change 1.47 1.12 31.6% p<.001 1.3times greater improvement

An advantage of:

Low Number of Users that Viewed resulted in nearly TEN times the gains in Math proficiency rates than the Districts average (p<.001)

Low Number of Users that Viewed resulted in more than 30% greater gains in Reading proficiency rates than the Districts average (p<.001)

But even fewer viewers outperformed no PD 360

9.7 times greater gains in Math vs. District

31.6% greater gains in Reading vs. District

23

Improvements in Proficiency Rates Correlated with PD 360

24

Sample Demographics State

State

TXTNMNKYFLCOAZAKP

erc

en

t

30

20

10

0

School Level

107 57.2 57.2 57.2

29 15.5 15.5 72.7

29 15.5 15.5 88.2

14 7.5 7.5 95.7

7 3.7 3.7 99.5

1 .5 .5 100.0

187 100.0 100.0

Elementary

Middle - JH (incl 6-8)

El-Mid (e.g. K-8)

High School

Mid-Hi (e.g. 6-12)

El-High (e.g. K-12)

Total

ValidFrequency Percent Valid Percent

CumulativePercent

• Eight States• 187 Schools

Improvements in Proficiency Rates Correlated with PD 360