Predictors of Success: Linking Student Achievement to School and Educator

Successes through On-Demand, Computer-Based Professional Learning

Steven H. Shaha, PhD, DBA Professor, Center for Public Policy & AdministrationIndependent Program Evaluator

Abstract

Year-over-year changes in student achievement were

analyzed for 734 schools selected due to utilization

history for online (i.e. on-demand) and computer-based

professional learning applications. Results showed that

schools with higher engagement in on-demand profes-

sional learning by educators significantly outperformed

their lower engagement counterparts in measures of

quantity and quality of utilization, participation, and

engagement. Higher engagement schools also had sig-

nificantly greater gains in student achievement as mea-

sured by percentages of students performing at profi-

cient or advanced levels. Higher engagement schools

also outperformed their lower engagement counter-

parts for gains in four key school- and educator-related

measures: teacher retention, dropout rates, student dis-

cipline issues, and rates of students with college-related

goals. Conclusions were that higher levels of utilization,

engagement, and active use are correlated with higher

student achievement and successes for both educators

and the schools in which they operate.

I. Introduction and Overview

Educators need high-impact help to keep their skills

well honed and to maintain their educational effec-

tiveness. Yet the body of literature linking professional

learning and development to gains in student perfor-

mance and teacher-related outcomes arguably remains

inadequate (cf. Shaha et al. 2004). Some studies have

shown that professional learning can lead to improved

student performance (cf. Garet et al. 2001; Desimone et

al. 2002; Shaha et al. 2004; Meiers and Ingvarson 2005;

Buczynski and Hansen 2010; Avalos 2011). Yet, it seems

clear from research that the more active an educator’s

participation is beyond traditional, passive profession-

al learning—such as sitting in a workshop or passively

watching a video alone—the greater the impact of par-

ticipation (cf. Garet et al. 2002; Desimone et al. 2002;

King 2002; Darling-Hammond 2004; Santagata, 2009).

In addition, roadblocks to teacher participation in pro-

fessional learning and implementation of skills learned

have been cited in recent research and remain import-

ant barriers to impact (cf. Buczynski and Hansen 2010).

Early research has begun to investigate the potential

benefits of computer-based and online professional

learning (cf. Farnsworth et al. 2002; Lewis et al. 2003;

Magidin et al. 2012; Rienties et al. 2013). However, at

least one recent study cited a “dearth of scientific

research … on whether changes in teachers’ knowl-

edge and instructional practices resulting from online

professional learning are linked to changes in students’

knowledge and practices” (Masters et al. 2012). We

also found that very few studies have investigated the

importance of leadership’s engagement in ensuring

the efficacy of professional learning, regardless of the

mode of delivery (cf. Sebastian and Allensworth, 2012).

Finally, there are education-related metrics that have

societal implications—metrics reflecting factors that are

critical in assessing the success of educators, schools,

and education as a societal institution. Despite the

importance of these assessments, we found virtually

no connection established in research between teacher

participation in professional learning and improvements

in student-related measures of non-test performance

beyond at-risk preschool children, such as dropout

rates or disciplinary rates (Wasik and Hindman 2011).

Even such seemingly simple correlations as improve-

ments in teacher retention (cf. Lathan and Vogt 2007)

or teacher attitudes and perceptions (cf. Guskey 2002)

resulting from professional learning are unaccountably

minimal in the research literature.

Taken as a whole, research indicates that providing edu-

cators with readily accessible learning opportunities has

a substantive and favorable impact. We relabeled this

approach as “on-demand learning” to accentuate why

it is effective instead of how it is delivered. One reason

for the effectiveness of the on-demand approach is that

educators learn about what they are most interested

in, or most in need of, at the time of interest or need,

rather than when it fits sequentially into any prescrip-

tive curriculum.

Thus timeliness of learning, synchronized with inter-

est and need, mean that educators benefit from what

could be labeled “just-in-time learning.” This concept

1Steven H. Shaha, PhD, DBA, Professor and/or Lecturer at University of Utah, Zayed University (UAE) and Harvard University

mirrors the business successes achieved in other in-

dustries through just-in-time approaches—or JIT—as a

science for maximizing efficiency and profitability while

minimizing costs associated with doing business (cf.

Bongiorni 2004; Hirano and Makota 2006; Ohno 1988;

Ruffa 2008). Education benefits from sciences proven

in industry further refine educator efficacy and its im-

pact on students. In this case, the near immediate and

personally customized benefits of online accessibility

provide for JIT educator learning: on-demand profes-

sional learning.

We undertook the designing and execution of an evalu-

ation study of on-demand professional learning in order

to answer a crucial set of inquiries regarding its impact

on students, educators, and schools. The driving re-

search questions therefore were whether schools with

higher utilization or engagement experienced greater

impact than those of lower utilization or engagement

for the following:

• Educator engagement in other metrics or areas of

utilization, participation, and engagement

• Student performance

• Other measures of school- and educator-related success

Additional questions to be addressed:

• Is viewing professional learning alone as strong a

predictor of success and impact as other metrics of

educator utilization, participation, and engagement?

• Is there a model or framework for predicting maxi-

mum impact from educator utilization, participation,

and engagement in on-demand or computer-based

teacher development applications?

II. Methods

A retrospective study was undertaken leveraging a

sample of 750 schools reflecting high engagement in

on-demand professional learning (i.e. PD 360® and

Observation 360®, School Improvement Network,

Salt Lake City, UT). Data included the 2009-2010 and

2010-2011 school years, categorized during analyses

as pre versus post. Schools were selected for inclusion

from the universe of on-demand users based upon

their active use as measured by minutes of viewing

professional learning videos, and minimum criterion for

inclusion was set at a minimum average of 90 minutes

per educator within any school, and all schools meeting

those minimum criteria were included. No inclusion or

exclusion criteria were implemented reflecting state or

district, rural or urban areas, school size or any other

variable associated with school or student demograph-

ics. Year-over-year improvement was computed as the

percentage change (i.e. gain or loss) for each metric

(i.e. [2011-2010]/2010]).

Educator Engagement: Levels of educator utilization,

participation, and engagement were loaded directly

from the on-demand applications as captured automat-

ically and transparently to users, thus ensuring objec-

tivity and accuracy, representing 27 metrics (further

explained in Results).

Student Performance: Performance data were gath-

ered from publically available sources. In order to

enable analyses across states with varying testing and

scoring approaches, data analyzed reflected the per-

centage of students classified as either proficient or

advanced on whatever approach applied within any

state or appropriate governing body. Data were limited

to reading and math only (2 metrics), as these were the

only two areas of measurement consistent across all

states. Sixteen schools were excluded from analysis for

inadequate data regarding student performance.

School-and Educator-Related Measures: A set of four

metrics were gathered by structured phone interviews

with each school, including rates for teacher retention,

dropouts, student discipline, and the number of stu-

dents reported as being college bound. Year-over-year

improvement was computed as the percentage change

(i.e. gain or loss) in the rate or percentage for each metric.

The final study included 734 schools in 211 districts

within 39 states. Schools were next classified into

quartiles reflecting their average minutes of use by

educator as a proxy for relative utilization or engage-

ment rates. To make analyses and conclusions more

straightforward for execution and interpretation,

analyses contrasted only the top and bottom quar-

tiles: the highest quartile of schools (higher engage-

ment schools) versus the lowest quartile schools

(lower engagement schools).

All analyses were conducted by an independent,

doctoral prepared, internationally recognized stat-

istician and program evaluator, using SPSS version

17.0 or higher, and SAS for confirmatory purposes as

needed or appropriate.

2

III. Results & Initial Interpretations

Viewed collectively, results showed that higher en-

gagement schools outperformed their lower engage-

ment counterparts in every area of measurement:

Educator Engagement: Higher engagement schools

outperformed their lower engagement counterparts

in 15 of the 27 metrics of utilization, participation, and

engagement, and performed equally well or better in

the remaining 12 metrics, although none significantly

(p>0.05). Higher engagement schools were significant-

ly higher in measures of implementation, accountability,

and oversight, or those metrics most appropriately as-

cribed to leaders and their role in successful execution

of the on-demand or computer-based, educator-learn-

ing program.

Metrics reflecting greater gains for higher engagement

schools included, for example, number of focus objec-

tives set up, observations performed, percent of regis-

tered users, and percent of users in communities. High-

er engagement schools performed significantly higher

in utilization metrics and measures of more passive

participation, including minutes viewed, forums viewed,

programs viewed, segments viewed, and links viewed.

In metrics classified as measures of engagement, higher

engagement schools outperformed the lower engage-

ment counterparts in metrics reflecting more active

engagement, including follow-up questions answered,

reflection questions answered, focus objectives set up,

forums posted, downloaded files, uploaded files, and

participation in communities.

Regarding the degree of comparative impact of

minutes viewed versus the other engagement met-

rics, higher engagement schools had 4.3% greater

minutes viewed (p<.01), a significant and important

utilization-related advantage (see Figure 1). This was

expected, since the assignment of schools to higher

and lower engagement categories was based upon

their comparative measures of viewing.

However, more revealing were the comparative

gains for higher versus lower engagement schools,

which were substantially and significantly greater for

utilization-related metrics, reflecting great levels of

active participation and engagement beyond simple

viewing. For example, we noted the magnitude of

difference in forums posted was 68.6% higher for

higher engagement schools (p<.001, see Figure 2),

illustrating that comparative gains were great against

lower engagement schools for measures reflecting

more active participation and engagement by educa-

tors. Similarly, the magnitude of difference in teacher

observations performed by leadership was 63.8%

higher for higher engagement schools (p<.001, see

Figure 3), illustrating that active engagement by

leadership was greater versus in lower engagement

schools, as well.

Figure 1. Comparative difference average minutes viewed per educator

Figure 2. Comparative difference in forums posted per educator

Figure 3. Comparative difference in observations performed per educator

13.7

23.1

0.0

5.0

10.0

15.0

20.0

25.0

Lower Engagement Schools

Higherer Engagement Schools

Forums Posted

20.6

33.7

0.05.0

10.015.020.025.030.035.040.0

Lower Engagement Schools

Higher Engagement Schools

Teacher Observations Performed

3

A complete view of the 15 measures for which high-

er engagement schools outperformed their lower

engagement counterparts is found in Table 1. For

convenience in interpretation of the results, the 15

metrics were categorized into logical groupings re-

flecting the apparent nature of the underlying con-

structs being measured. The grouping labeled Lead-

ership, Implementation, and Accountability included

metrics reflecting program setup and active leader

engagement. The Educator Utilization grouping in-

cluded metrics reflecting the more passive measures

of participation as contrasted with educator engage-

ment, for which the metrics reflected more active

and productive participation, for example, beyond

viewing alone.

Table 1. Comparative performance in measures of educator

participation as categorized

The implication was that video viewing alone, or other

more passive metrics (e.g. % users registered), were

not as great of predictors or discriminators of educator

engagement as were measures of utilization reflecting

more active engagement.

Student Performance: Higher engagement schools

collectively began at a significant performance disad-

vantage in both reading (p<.001) and math (p<.001) in

terms of the percentage of students classified as either

proficient or advanced. However, in reading, higher en-

gagement schools successfully closed the performance

gap with the lower engagement schools (see Figure 4).

While the lower engagement schools improved by an

impressive 4.9% year over year (p<.001), the higher en-

gagement schools improved by 18.0% (p<.001), nearly

four times the rate of improvement comparatively.

In math, higher engagement schools not only closed

the pre-existing performance gap, but significantly

surpassed the lower engagement schools year over

year (p<.001, see Figure 5). Lower engagement schools

did experience improvement from on-demand profes-

sional learning at 0.5% year over year (p<.05). However,

the higher engagement schools improved by 18.9%

(p<.001), over 30 times the rate of improvement com-

paratively. Interestingly, this rate of improvement very

nearly equaled the rate achieved in reading.

Figure 4. Comparative gains in reading performance

Figure 5. Comparative gains in math performance

School-Related Engagements: Results revealed statis-

tically significant relationships between key metrics of

educator/school-related success and higher and more

active utilization of the on-demand professional learn-

ing. Both higher and lower engagement school cohorts

saw statistically significant gains in school-related

metrics. However, higher engagement schools, which

were consistently those with higher utilization rates for

on-demand professional learning, also achieved better

improvement year over year versus lower engagement

Higher Engagement

Schools

Lower Engagement

Schools DifferencePercent

DifferenceLeadership, Implementation, AccountabilityFocus Objectives Set Up 130.1 29.2 100.9 345.5%Observations Performed 3120.7 2149.0 971.7 45.2%Percent Registered Users 87.8% 83.2% 0.0 5.5%Percent of Users in Communities 43.0% 36.5% 6.5% 17.8%

Educator UtilizationMinutes Viewed 359.9 80.6 279.3 346.5%Forums Viewed 138.7 87.0 51.7 59.4%Programs Viewed 588.7 223.5 365.2 163.4%Segments Viewed 2298.0 528.9 1769.1 334.5%Links Viewed 12.2 10.6 1.6 15.1%

Educator EngagementFollow-up Questions Answered 359.6 167.0 192.6 115.3%Reflection Questions Answered 588.3 340.4 247.9 72.8%Focus Objectives Set Up 3120.7 2149.0 971.7 45.2%Forums Posted 28.5 23.4 5.1 21.8%Downloaded Files 45.2 35.4 9.8 27.7%Uploaded Files 46.3 29.3 17.0 58.0%Participation in Communities 43.0% 36.5% 6.5% 17.8%

63.566.6

56.9

67.2

50.052.054.056.058.060.062.064.066.068.0

Pre Post

Percent of Students Proficient or Advanced:Reading

Lower Engagement Schools

Higher Engagement Schools

62.7 63.0

58.4

69.5

56.058.060.062.064.066.068.070.072.0

Pre Post

Percent of Students Proficient or Advanced:Math

Lower Engagement Schools

Higher Engagement Schools

4

schools in every measure of educator- and school-relat-

ed success available, including the following:

• 20.0% lower dropout rates (p<.001) versus 4.9%

lower dropout rates for the lower engagement

schools (p<.01), representing 4-times greater in gains

(see Figure 6)

• 9.6% gain in rate of students with goals to attend col-

lege (p<.001) versus flat gains for lower engagement

schools (p=ns), or 12-times the gains

• 33.2% lower rate for student discipline occurrences

(p<.001) versus 7.4% lower for the lower engagement

schools (p<.01), greater than 4-times the gains

• 2.8% higher teacher retention rates (p<.001) versus

1.7% lower for the lower engagement schools (p<.01),

nearly 2/3 greater gains

Figure 6. Comparative gains/reductions in dropout rates

IV. Discussion & Conclusions

Results substantiated significant and substantive

advantages to the use of on-demand and comput-

er-based professional learning. Further, results clearly

indicated that the more engaged the user is beyond

video participation alone, the greater the impact of the

professional learning. All five research objectives were

achieved.

Higher engagement schools—those with higher uti-

lization and participation—outperformed their lower

engagement counterparts in the majority of the metrics

analyzed, and never underperformed on any other met-

rics. Higher engagement schools experienced signifi-

cantly greater gains for students in math and reading,

equaling or exceeding 18% gains over prior-year per-

formance levels. For school-related measures, higher

engagement schools experienced significantly great-

er gains in critical measures of educator and school

success, including teacher retention, student discipline,

dropout rates, and the number of students reported as

college bound.

Additionally, video viewing alone was not as great an

indicator of student and educator- and school-relat-

ed gains as the other host of utilization, participation,

and engagement metrics. Performance on the other

metrics of engagement far exceeded those found for

video viewing alone, generally by magnitudes of 10 to

20 times. Several interpretations might fit to explain

the finding, perhaps best expressed as questions: Does

video viewing alone possibly include multi-tasking by

the participants who may be reading, updating grade

books, or emailing while videos are streaming? Do

metrics of more active participation reflect higher levels

of personal engagement, and therefore more active

learning and focus and higher likelihood of application

of things learned? While other explanations may apply,

these data support conclusions that the gains achieved

for students, educators, and school-related impacts

support leveraging professional learning programs that

go far beyond video watching alone.

Taken as a whole, results reveal significant predictive

correlations between the quantity of educator utiliza-

tion, participation, and engagement with better student

results and school-related outcomes. While correlation

cannot prove causation, the systematic and consistent

findings within these data clearly support a conclu-

sion that participation and engagement in this form of

teacher development resulted in the advantages and

gains found. It is intuitive that highly active and more

frequent participation in professional learning should

lead to educators more focused on critical behaviors

and techniques that would help them teach better, and

help their students achieve more proficiency.

Higher, more frequent on-demand activity, combined

with the perception of improved student success

before quantified, also resulted in the higher teacher

retention rates observed, another indicator of educator

satisfaction and greater enthusiasm for teaching. While

it could be argued that retention is a prescient indicator

of improved success—better teachers stay—the oppo-

site causal perspective is at least equally supported in

this study. In each measurement area, higher engage-

ment schools outperformed the corresponding lower

engagement cohort, even when student performance

began at comparatively lower levels. Thus, the data

support the conclusion that higher engagement in

5.35.0

5.1

4.1

3.0

3.5

4.0

4.5

5.0

5.5

Pre Post

Dropout Rate

Lower Engagement Schools

Higher Engagement Schools

5

on-demand learning results in higher teacher percep-

tions of their impact on students, contributing to higher

satisfaction, and ultimately higher retention.

The next phase of research should include a repeat of

these analyses for data reflecting the 2011-2012 school

year. That may include a contrast of schools for which

the 2011-2012 was the first year of participation versus

the gains found in these year-to-year analyses for 2010

versus 2011. Additionally, new analyses might contrast

the newly implemented schools for 2012 with a two-

year history of gains for those included in these anal-

yses, thus adding the insight of multiple year-on-year

participation from 2010 through 2012 versus a first year

alone.

Future research might also include contrasts of gains

with changes experienced in equivalent non-participat-

ing schools to ensure that the accomplishments found

here reflect achievements above and beyond chance

alone. This would be ideally executed with a study that

includes analyses of changes contrasting schools with

and without the on-demand or computer-based teach-

er learning, thus controlling for important underlying

differences such as socioeconomic mixes. Additional

research might also include the contrasting metrics

from higher utilization schools with non-participating

schools from their respective districts, thus controlling

for differences in demographics and socioeconomics.

References

Ava los, Beatrice. “Teacher Professional Development in Teaching

and Teacher Education Over Ten Years.” Teaching and Teacher

Education 27, no. 1 (1, 2011): 10-20.

Bon giorni, Sara. “All in the Timing.” The Greater Baton Rouge Busi-

ness Report (July 19, 2004).

Buc zynski, Sandy, and C. Bobbi Hansen. “Impact of Professional

Development on Teacher Practice: Uncovering Connections.”

Teaching and Teacher Education 26, no. 3 (2010): 599-607.

Dar ling-Hammond, L. “Standards, Accountability, and School Re-

form.” Teachers College Record 106, no. 6 (2004): 1047-85.

Des imone, L. M., A. C. Porter, M. S. Garet, K. S. Yoon, and B. F.

Birman. “Effects of Professional Development on Teacher’s

Instruction: Results from a Three-Year Longitudinal Study.” Ed-

ucational Evaluation and Policy Analysis 24, no. 2(2002): 81-112.

Far nsworth, B. and et al. “Preparing Tomorrow’s Teachers to use

Technology: Learning and Attitudinal Impacts on Elementary

Students.” Journal of Instructional Psychology 29, no. 3(2002).

Gar et, Michael S., Andrew C. Porter, Laura Desimone, Beatrice

F. Birman, and Kwang Suk Yoon. “What Makes Professional

Development Effective? Results from a National Sample of

Teachers.” American Educational Research Journal 38, no. 4

(Winter 2001, 2001): 915.

Gus key, Thomas R. “Professional Development and Teacher

Change.” Teachers and Teaching 8, no. 3 (08/01; 2012/11,

2002): 381-91.

Hir ano, Hiroyuki, and Furuya Makota, eds. JIT is flow: Practice and

Principles of Lean Manufacturing. ISBN 0-9712436-1-1 ed.: PCS

Press, Inc., 2006.

Kin g, Kathleen P. “Identifying Success in Online Teacher Education

and Professional Development.” The Internet and Higher Edu-

cation 5, no. 3 (0, 2002): 231-46.

Lat ham, N., and W. P. Vogt. “Do Professional Development Schools

Reduce Teacher Attrition? Evidence from a Longitudinal Study

of 1,000 Graduates.” Journal of Teacher Education 58, no. 2

(2003): 153-67.

Lew is, V. K., S. H. Shaha, B. Farnsworth, L. Benson, and D. Bahr.

“The Use of Assessment in Improving Technology-Based

Instruction Programs.” Journal of Instructional Psychology 30,

no. 2 (2003).

Mag idin de Kramer, Raquel, Jessica Masters, Laura M. O’Dwyer,

Sheralyn Dash, and Michael Russell. “Relationship of Online

Teacher Professional Development to Seventh-Grade Teachers’

and Students’ Knowledge and Practices in English Language

Arts.” The Teacher Educator 47, no. 3 (06/27; 2012/11, 2012):

236-59.

Mas ters, Jessica, Raquel Magidin de Kramer, Laura O’Dwyer, Sher-

alyn Dash, and Michael Russell. “The Effects of Online Teacher

Professional Development on Fourth Grade Students’ Knowl-

edge and Practices in English Language Arts.” Journal of Tech-

nology and Teacher Education 20, no. 1 (January, 2012): 21-46.

Mei ers, M. and L. Ingvarson. “Investigating the Links Between

Teacher Professional Development and Student Learning Out-

comes.” Australian Council for Educational Research 1, (2005):

1-93.

Ohn o, Taiichi, ed. Just-in-time for today and tomorrow. ISBN

0-915299-20-8 ed.: Productivity Press, 1988.

Rie nties, Bart, Natasa Brouwer, and Simon Lygo-Baker. “The Ef-

fects of Online Professional Development on Higher Education

Teachers’ Beliefs and Intentions Towards Learning Facilitation

and Technology.” Teaching and Teacher Education 29, no. 0 (1,

2013): 122-31.

Ruf fa, Stephen A. “Going Lean: How the Best Companies Apply

Lean Manufacturing Principles to Shatter Uncertainty, Drive In-

novation, and Maximize Profits.” AMACOM (American Manage-

ment Association) (2008).

San tagata, R. “Designing Video-Based Professional Development

for Mathematics Teachers in Low-Performing Schools.” Journal

of Teacher Education 60, no. 1 (2009): 38-51.

Seb astian, J. and E. Allensworth. “The Influence of Principal Lead-

ership on Classroom Instruction and Student Learning: A Study

of Mediated Pathways to Learning.” University Council for

Educational Administration 10, (2012): 626-63.

Sha ha, S. H., V. K. Lewis, T. J. O’Donnell, and D. H. Brown. “Evaluat-

ing Professional Development: An Approach in Verifying Pro-

gram Impact on Teachers and Students.” Journal of Research

in Professional Learning 1, no. 1 (2004): 1.

6