Virtual School Symposium 2008
Evaluation in K-12 Online LearningLessons from the Field
Evaluation in K-12 Online Learning
Presenters Cathy Cavanaugh , University of
Florida Martha Donaldson, Alabama
ACCESS Mickey Revenaugh, Connections
Academy Donna Scribner, VHS Inc.
Moderator: Tom Clark, TA Consulting
Special Thanks to Brian Lekander,OII
Evaluation in K-12 Online Learning
AgendaIntroductionsPublication Overviews: OII Evaluating Online Learning Guide NACOL Issue Brief (in process)
Lessons from the Field: Preparing to EvaluateConducting Effective EvaluationsUsing Evaluation Results
Q & A
Evaluation in K-12 Online Learning
Office of Innovation and ImprovementU.S. Department of Education
.
Evaluating Online
Learning: Challenges
and Strategies for
Success
The latest guide in the Innovations in Education series
Office of Innovation & Improvement, U. S. Department of
Education
Evaluation in K-12 Online LearningWhy Did OII Prepare This Guide? Continued skepticism in some
quarters about quality of online learning
Desire for greater accountability in Gov’t funding programs
Concern that innovation often outpaces what we know about educational technology
Evaluators of federally-funded projects are sometimes unfamiliar with online technologies
Evaluation in K-12 Online LearningGuide Based on Seven Case Studies Alabama ACCESS Algebra I Online (Louisiana) Appleton eSchool (Wisconsin) Arizona Virtual Academy Chicago Public Schools Virtual
High School Digital Learning Commons
(Washington) Maryland Public Television’s
Thinkport
Evaluation in K-12 Online Learning
Who is this Guide For? Online program administrators
who need to think strategically about evaluation – and how they will use it as their program evolves
Evaluators who are not very experienced with online learning and the challenges it presents
Evaluation in K-12 Online LearningWhat is the Guide’s Focus? Evaluation Challenges Commonly
Encountered in Online Evaluations
Instructive Examples of Responses to these Challenges
Evaluation in K-12 Online Learning
Six Evaluation Challenges Featured Meeting the Needs of Multiple
Stakeholders Building on the Existing Base of
Knowledge Evaluating Multifaceted Online
Resources Finding Appropriate Comparison Groups Interpreting the Impact of Program
Maturity Translating Evaluation Findings into
Action
Evaluation in K-12 Online LearningSome Things OII Has Learned Evaluation of online learning is
inherently political Programs will always be pressed for
information/results before they would like to be
It’s best to be proactive by anticipating the needs of your stakeholders
You won’t always be able to control the spin on your results
It’s helpful to think of evaluation as an ongoing process
Evaluation in K-12 Online Learning
Where Can I Get the OII Guide?
To view online: http://www.ed.gov/about/pubs/intro/innovations.html
From Ed Pubs:Order online at http://www.edpubs.gov or call 1-877-433-7827 (request order number ED004344P)
Evaluation in K-12 Online Learning
Office of Innovation and ImprovementU.S. Department of Education
.
Quality and Effectiveness in Online Learning
Issues Brief
Research Committee
North American Council for Online
Learning
Evaluation in K-12 Online LearningNACOL Research Committee Quality and Effectiveness in K –
12 Online Teaching Research Based Practices Practices currently adopted by
Online course providersPractices and Policies for K-12
Online Teaching and LearningOnline Professional Development
Standards Across North AmericaFeatures of Teaching in Virtual
Schools
Evaluation in K-12 Online LearningNACOL Research Committee Online Teacher Support Programs:
Mentoring and Coaching Models Description of the Mentoring/Coaching
relationships in vignettes from perspectives of several virtual high schools
Alabama ACCESS Distance Learning Colorado Online Learning Florida Virtual School Idaho Digital Learning Academy Mississippi Virtual School Missouri Virtual Instructional Program Tennessee: e4TN Virtual High School Global Consortium
Evaluation in K-12 Online Learning
Introduction Why evaluate online learning? To demonstrate the value or
worth of your program To improve your program over
time To document participant
outcomes related to program goals
To meet stakeholder interests/accountability needs
Evaluation in K-12 Online Learning
Introduction What happens when evaluation is
neglected? Program set up without clear goals Focus on activities and simple outputs Desired change in participant outcomes
undefined Data essential to studying success
remains undefined, ungathered Focus on anecdotal evidence,
testimonials Stakeholder information needs
neglected Program unable to demonstrate value
or worth
Evaluation in K-12 Online Learning
Preparing to Evaluate Question posed to panelists
(program managers & evaluators):Please think back to the start of the online learning program. What did you do to prepare to evaluate quality, effectiveness, and impact?
Question for current programs: How can you set up your online learning program to study quality & effectiveness over time?
Evaluation in K-12 Online Learning
Alabama ACCESS: Look at Your Specific Objectives
To Provide:
Equal Access to High Quality Instruction An Infrastructure That Delivers Quality
Learning Opportunities Greater Equity for all Alabama Public High
School Students Through Cutting-Edge Technology
Wide Range of Courses Available to Relatively few Alabama Students Today (“Advanced Diploma” Courses, Advanced Placement Courses, Additional Course Offerings, Remediation and Supplemental Resources)
Preparing to Evaluate
Evaluation in K-12 Online Learning
ACCESS—Established Need for Evaluation—Quality, Effectiveness, Impact
Built Into Program Design Purpose
Provide Evidence that Goals of Initiative are Met
Gauge Satisfaction with Courses Recommend Changes in Procedures and
Resources to Improve and Strengthen Program and Increase its Positive Impact
Determine Effectiveness of Regional Support Centers
Preparing to Evaluate
Evaluation in K-12 Online Learning
ACCESS—Preparation for Evaluation
Development of RFP at Beginning of Initiative
Development of Evaluation Plan with ISTE Identification of:
Questions to be AnsweredData that will be NeededData that will not be AvailableEvaluation Methods to be Used
Development of Data Collection Instruments
Preparing to Evaluate
Evaluation in K-12 Online Learning
ACCESS—What Worked
Financial support from the state. Rapid expansion of the infrastructure. Highly rated Regional Support
Centers. Thousands of students take classes
that would be otherwise unavailable. Teaching practices change with
increased technology integration and student-centered pedagogy.
Preparing to Evaluate
Evaluation in K-12 Online Learning
ACCESS—Strategy-in-Progress
Seek out teachers with virtual and F2F students.
Compare outcomes for students in similar classes with the same teacher.
Preparing to Evaluate
Evaluation in K-12 Online Learning
ACCESS—Evaluation Issues
Availability of data (e.g., lack of common end-of-course tests).
Access to data (e.g., Advanced Placement records held by testing vendor).
Technological barriers (e.g., firewalls prevent submission of survey responses).
Research design challenges.
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium
Mission To develop and deliver standards
based student centered online courses to expand students’ educational opportunities and 21st century skills and to offer professional development to teachers to expand the scope and depth of their instructional skills.
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Believes that:• Student-centered online courses can
be designed and delivered to students to promote a high quality collaborative learning environment.• in which student exchange and interaction is
a valued component of the instructional process.
• Educational opportunity need not be limited by barriers of time/place/lack of qualified faculty.
• Rather, we believe that high-quality education is possible-today-for all students in all locations.
• Online education offers any school with Internet connectivity a wealth of trained, experienced faculty members qualified in numerous disciplines, for teaching a wide array of courses designed to meet the needs of all students. An innovative, standards-based curriculum delivered online offers diverse, exciting learning choices for students, and the opportunity and skills to participate in a national and global community.
Preparing to Evaluate
Evaluation in K-12 Online Learning
• Online teaching should augment rather than replace traditional classroom teaching. • The Virtual High School's online courses are a
proven, flexible solution for schools needing an expanded curriculum, teachers seeking new horizons, parents wanting more involvement with their children's education, and a society grappling with ways to offer opportunity to all its citizens.
• The goals of education are advanced best by putting value and service first. • When schools work together in a collaborative
network such as VHS, they become part of an abundant and generous educational community that promotes the affordable sharing of professional resources
VHS Believes that:
Evaluation in K-12 Online Learning
VHS Global Consortium 1996 – Technology Innovation
Grant 5-year, $7.4 million US Department of Education
Non-profit; non-degree granting Consortium of schools
575+ member schools 30 states; 39 countries 11,000+ students
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium Program Evaluation
Outsider reviewer Quality, Growth, Program Goals Surveys Superintendents, Principals,
VHS Teachers, VHS Students, VHS Site Coordinators
Published Annually Available to Public via Website
(www.goVHS.org) Quality Benchmark Indicators
(QBIs)
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium
QBIs Growth Indicators measured
against Growth Goals Quality Indicators – tie back to
Mission & Beliefs Quality of Courses Quality of Professional Development Quality of Services & Program
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium
Course Quality Benchmark Indicators
0%
20%
40%
60%
80%
100%
00/01 01/02 02/03 03/04 04/05 05/06 06/07
AP exam pass rate
AP exam take Rate
Course completion rate
Credit recovery rate
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium
Course Completion Rate over Enrollments
0%10%20%30%40%50%60%70%80%90%
100%
00/01 01/02 02/03 03/04 04/05 05/06 06/07
0
2000
4000
6000
8000
10000
Course completion rate
VHS Students
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium
PD Quality Indicators - First-year Teachers
0%
20%
40%
60%
80%
100%
02/03 03/04 04/05 05/06 06/07 07/08
PD Graduation Rate
PD EffectivenessRate
Preparing to Evaluate
Evaluation in K-12 Online Learning
VHS Global Consortium
Program Services Quality Benchmark Indicators
65%
70%
75%
80%
85%
90%
95%
02/03 03/04 04/05 05/06 06/07
Membership RetentionRate
Course Seat UtilizationRate
Preparing to Evaluate
Evaluation in K-12 Online Learning
Connections Academy Connections Academy programs
are mostly full-time and include K-8 Unique research challenges:
Seeking data from younger children and parents; no additional “program ally” such as site facilitator
Unique research benefits: Address whole learner, gather all demographics, include state test results/NCLB data
Preparing to Evaluate
Evaluation in K-12 Online Learning
Connections Academy Built into Connections
Academy program: SIS data within our LMS Data analysis: Data views Log: Teacher communication, action Parent Satisfaction Survey StarTracker: Embedded feedback
on every lesson plus school as a whole
Measurable school and company goals
Preparing to Evaluate
Evaluation in K-12 Online Learning
Lessons Learned/Next Steps What are some lessons learned
about effective practices in evaluating online learning?
What can we learn from the results of an evaluation?
How can we use results to improve the program? To inform stake-holders & decision-makers?
Evaluation in K-12 Online Learning
Lessons Learned/Next Steps Some Lessons Learned (OII,
2008) – Evaluations should: Effectively inform stakeholder groups Share tools and research methods Focus on outcomes, not activities Recruit willing research populations
early Obtain data access, or plan to gather it Move from formative to summative Disseminate timely information to
internal & external decision-makers
Evaluation in K-12 Online Learning
Lessons Learned/Next Steps What can we learn from an
evaluation? Satisfaction measures Quality & effectiveness measures Changes in knowledge & skills via
participation (participant outcomes) How can we use results to
improve the program? To inform stakeholders & decision-makers?
Evaluation in K-12 Online Learning
Lessons Learned/Next Steps Question posed to panelists
(program managers and evaluators): Based on your experience using or
conducting evaluations of online learning programs, please share:
1) What are some lessons you’ve learned about effective evaluation practices?
2) How has evaluation helped you improve an online learning program or demonstrate its worth?
Evaluation in K-12 Online Learning
Connections Academy When Evaluation Pays Off
Early MoVIP K-5 Results: User satisfaction high, teachers make the difference – validation of model
Mississippi K-8 Pilot: Even a small, short pilot can be positively revealing if designed with evaluation in mind
Ongoing Parent Satisfaction Surveys: Overall high rates (90%+) persist, and improvements in “iffy” areas absolutely trackable
Lessons &Next Steps
Evaluation in K-12 Online Learning
Connections Academy Lessons Learned
No substitute for familiarity: Evaluators need to be equipped to really dig into curriculum and platform
Data transmission is an art: Foster friendships between the evaluators and the program data wonks
Positive results are no guarantee: As in Mississippi example – can’t make up for lack of support
Lessons &Next Steps
Lessons &Next Steps
Evaluation in K-12 Online Learning
Connections Academy Lessons Learned
Transparency takes some getting used to: Educators are not accustomed to having practice so visible
Evaluation is only half the battle: True continuous improvement takes precision and persistence
Patience is a virtue: User satisfaction and academic results may diverge in beginning, but will converge if students stay
Lessons &Next Steps
Evaluation in K-12 Online Learning
VHS—Lessons Learned Don’t make decisions on a snap-
shot in time… use longitudinal data and look for trends
Evaluation criteria derives from Objectives; Objectives derive from Mission
It may be interesting but does it inform?
Lessons &Next Steps
Evaluation in K-12 Online Learning
VHS—Lessons Learned 2Continuous Course
ImprovementStudentProfessional Development
Need for Teacher SupportProgressElluminate SessionsFaculty Advising on graduated
scale
Lessons &Next Steps
Evaluation in K-12 Online Learning
How has the evaluation helped to improve the ACCESS program?
Year I Findings
Course Changes Needed
Implications/Improvements
Course Revisions Made Increased Use of Voice
Tools, Addition of Improved Speaking Assignments and Examples, and Use of Headphones in Foreign Language Courses
More Detailed Alignment and Gap Analysis Process
Addition of Course Development/Revision Component
Lessons &Next Steps
Evaluation in K-12 Online Learning
Year I Findings
Need for Better/Increased Communication and Interaction Among Teachers, Students, Facilitators, and Support Center Staff
Implications/Improvements Introduction of Regular
Faculty Meetings via Web Conferencing
Assignment of SDE Liaison to Each Support Center Region
Expansion of Contract for Web Conferencing Capability
Additions to Training Agenda Review of Teacher Pay Issues
How has the evaluation helped to improve the ACCESS program?
Lessons &Next Steps
Evaluation in K-12 Online Learning
Year I Findings
Need for Additional Professional Development and Training Modules
Implications/Improvements Development of Additional
Training Modules Modification of
Professional Development Plan
Addition of SDE Staff Member to Coordinate Professional Development
Establishment of Teacher Mentoring Plan
Development of C.A.S.T. Site for Teachers
How has the evaluation helped to improve the ACCESS program?
Lessons &Next Steps
Evaluation in K-12 Online Learning
Year I Findings Need for Assistance
With Scheduling, Registration, and Enrollment Issues
Increased Number of Students Not Prepared/Ready for Assigned Class
Implications/Improvements
Development of Training Module for Counselors on the Registration Process
Expansion of Meetings with Counselors (Regional and State Meetings)
Decision to Develop a New Student Registration Site
Onsite Visits and Individualized Telephone Calls to Assist With Process
Further Look at Course Prerequisites
How has the evaluation helped to improve the ACCESS program?
Lessons &Next Steps
Evaluation in K-12 Online Learning
Year I Findings
Need for Assistance With Technical Issues
Implications/Improvements
Use of SDE and Regional Support Center Helpdesks
Additional School Visits Addition of Staff at SDE Equipment/Connectivity
Checks by SDE, Support Centers, and Alabama SuperComputer Authority
Increased Communication With School Staff
Identification of Key Areas of Concern
How has the evaluation helped to improve the ACCESS program?
Lessons &Next Steps
Evaluation in K-12 Online Learning
Year I Findings
Need for Increased Number of Responses on Interviews and Surveys
Implications/Improvements
Identification of Reasons for Poor Response Rate Timing Surveys Used Filters/Blocks at
School Level Content/Clarity
Issues
How has the evaluation helped to improve the ACCESS program?
Lessons &Next Steps
Evaluation in K-12 Online Learning
Access—Recommendations for Future
• Increase Teacher Training, Including Additional Hands-on Time With ACCESS Technologies
• Further Develop Ways to Ensure High Interaction Between Teachers and Students (Such as Moving up of Timeline for Blended Model)
• Increase Role of Support Centers in Providing Training (More Targeted Training for Counselors and Facilitators)
• Continue Annual Audits of Support Centers• Continue emphasis on What Data is Available and
How it Can be Obtained• Ensure That New Student Registration & Data
System is Designed to Collect Needed Data and to Integrate Effectively with LMS for Data Collection and Reporting
Lessons &Next Steps
Evaluation in K-12 Online Learning
Q & A Time
Do you have questions for our panelists about evaluating online learning?
PPT slides will be posted at:vss2008.wikispaces.com/
field
Evaluation in K-12 Online Learning
Thank You! Thanks for your participation in
our session today!
Tom Clark (moderator)Cathy CavanaughMartha DonaldsonMickey Revenaugh
Donna ScribnerThanks to OII & NACOL
Top Related