A Holistic Approach to Assessing eLearning A Holistic Approach to Assessing eLearning Dr. Tom...
-
Upload
jonathan-kristian-gibbs -
Category
Documents
-
view
216 -
download
0
Transcript of A Holistic Approach to Assessing eLearning A Holistic Approach to Assessing eLearning Dr. Tom...
A Holistic ApproachA Holistic Approach toto
Assessing eLearningAssessing eLearning
Dr. Tom Wilkinson, DirectorThe Institute for
Distance and Distributed LearningVirginia Tech
Appalachian College AssociationSummit 2004
Abingdon, Virginia
eLearning @ Virginia Tech…. excellence and access
• Nearly 60,000 credit and non-credit eLearning enrollments since Su98.
• 1,672 eLearning course offerings since Su98.
• 568 different eLearning courses offered in past 5 years
• 21 master degrees/certificate/ re-certification/licensure programs
• 85% of academic departments participating
• 89% of online learners were more confident they could reach their academic goals
• 88% rated the quality of instruction good or excellent
0
100
200
300
400
500
600
93/ 94 95/ 96 97/ 98 99/ 00 01/ 02 03/ 04
Growth in eLearning course offerings
Created in 1999, the Institute for Distance and Distributed Learning provides leadership, coordination, management and support to the distance and distributed learning (eLearning) activities of Virginia Tech. The Institute takes a holistic approach to distance learning where the students total educational experience is considered.
Electronically extend the campus
Provide an Open Campus
Research& Development
International Education
Specialized& Contractual
Virginia Tech’s ... Institute for Distance & Distributed Learning
AcademicAdministrative Support
Student & Faculty
Academic
A Holistic Approach to eLearning… the whole of the experience is greater than the sum of its parts
Information Systems
University Engagement … of distance and distributed learning at Virginia Tech
Bursar
Faculty
Library
Alumni
Enrollment Services
CEUT/CIL
Research
University Outreach
Budget &Finance
AdministrativeServices
GraduateSchool
IDDL
Colleges & Departments
Academic
StakeholdersFaculty
Cost
Access
Learning
Systems
Student
Stakeholders
A Holistic Model of eLearning … the whole of the experience is greater than the sum of its parts
Information Systems
Colleges & Departments
University Engagement … of distance and distributed learning at Virginia Tech
Bursar
Faculty
Library
Alumni
Enrollment Services
CEUT/CIL
Research
University Outreach
Budget &Finance
AdministrativeServices
GraduateSchool
Access
CostSystems
Faculty
Student Learning
Stakeholders
• 89% increase in eLearning enrollments• 37% increase in academic departments involvement• 20% of eLearning faculty are honored faculty• 151% increase in eLearning courses • 120% increase in degree & certificate programs • 100% increase in high enrollment online courses• 34% increase in centers & sites served
• 95% of barriers eliminated or significantly reduced • Nearly $2 million in IDDL Enterprise Fund eligible revenue • Award winning programs & the Chronicle article
eLearning at Virginia Tech…1998-2004
Assessment of Online Learners
• 91% were satisfied or very satisfied
• 96% rated services about the same, better or much better than with traditional courses
• 89% were more confident they could reach their academic goals
• 89% said they could not have taken this course in a traditional format
• 92% completion rate of “virtual” students
…over 1,500 respondents
SO WHAT…does that tell us about the quality of our efforts?
Why Assess eLearning?
Who Assesses eLearning?
What about eLearning do we assess?
When do we assess eLearning?
What is Quality eLearning?
“Quality is hard to define,
but I know it when we see it.”
Holistic Approach to eLearning Quality The whole of the experience is greater than the sum of its parts
Academic
Stakeholders
Institute for Distance and Distributed Learning
Faculty Satisfaction
Cost Effectiveness
Access
Learning Effectiveness
System Effectiveness
Student Satisfaction
Stakeholders
What are the Key Indicators of Quality?
• Accrediting agencies (SACS)
• State/Federal Agencies (SCHEV, DoE)
• Best Practices (WCET, ACHE)
• Sloan-C “Quality Pillars”
• Institutional Requirements
• Operational Requirements
eLearning Quality Matrix
Key Quality Indicator Documentation– KQI Source– Quality Metric– Process/Practice– Documentation– Assessment Schedule– Score Card
The KQIs….Key Quality Indicators
Distance and Distributed Learning at Virginia TechKey Quality Indicators
Key Quality Indicators
Criteria Process DocumentationAssessment
Schedule Metrics
Score
1. Institutional Indicators
1a. The program is consistent with the institution’s role and mission
- SACS criteria- SACS Best Practices- ACHE Guidelines- WCET Principles
Review for consistency at yearly IDDL strategic planning retreats
- University Strategic Plan- Academic Affairs Strategic Plan- DDL Strategic Plan
May each year
IDDL Mission and Strategic Plan is consistent with University Mission and Strategic Plan
3
1b. Compliance with SACS notification accrediting requirements
- SACS Criteria- SACS Best Practices- WCET Principles
Review of program offerings with SACS notification file
- VT SACS notification process.- Grad. School program list.- SACS notification file.
Juneeach year
SACS notified of all appropriate distance learning program changes.
3
1c. University’s budgets and policy statements reflect its commitment to Learners
- SACS Accreditation- SACS Best Practices- WCET Principles- ACHE Guidelines
Review university budget for DDL inclusion and IDDL budget for adequacy; Review policy statements for DDL
- University Strategic Plan- DDL strategic Plan- University Budget- IDDL budget allocation- IDDL Enterprise Fund report
July each year
IDDL budget on university base budget;
2
1d. Adequacy of technical & physical plant facilities including appropriate staffing and technical assistance
- SACS Accreditation- SACS Best Practices- WCET Principles- ACHE Guidelines
Conduct student & faculty course and support service evaluations;Review technology plan and budget
- student evaluations- faculty evaluations- support staff evaluations
End of each term
At least 85% of all respondents satisfied with technology and technical assistance;Budget plan provides for technology updating; Staffing structure is appropriate
3
Example 1
Key Quality Indicators Criteria Source (s) Process
1. Institution/Organization Indicators1a. The program is consistent with the institution’s role and mission.
• SACS criteria
• SACS Best Practices
• ACHE Guidelines
• WCET Principles
Review for consistency at yearly IDDL strategic planning retreats
Example 1 (continued)
DocumentationAssessment
Schedule Metric (s) Score
• University Strategic Plan
• Academic Affairs Strategic Plan
• DDL Strategic Plan
• IDDL Strategic Plan
May / June
each year
Program is consistent with University Mission and Strategic Plan
3
Score: 3 = meets or exceeds Key Quality Indicator 2 = meets most of KQI 1 = meets some of KQI 0 = does not meet KQI
Example 2
Key Quality Indicators Criteria Source (s) Process
4. Student Indicators4a. Appropriate services are available for students of electronically offered programs.
• SACS criteria
• SACS QEP
• SACS Best Practices
• ACHE Guidelines
• WCET Principles
Identify appropriate online student services through:
• consultation with university student affairs division
• student surveys
• utilization reports
• evaluation of services
Example 2 (continued)
DocumentationAssessment
Schedule Metric (s) Score
• Student Services for Distance Learners thesis results
• Online Wellness Resource Center dissertation results
• Hierarchy of Student Services needs for distance learners (independent study)
• Course evaluations
• OWRC utilization results
• Yearly identification of service needs
(May/June)
• Student satisfaction & service utilization each semester
•Online student services provided if 10% or more of students indicate the importance of service.
•90% or more of distance learners rate services good or excellent.
3
Contextually Designed Models… addressing changes within the system
Pedagogical Considerations- course management- opportunity for interaction and engagement- extending research- real-world application- assessment and evaluation- content life-cycle
Economic Considerations- business plan- costing break points- program maintenance including course re-design/development- faculty and department incentives- revenue distribution- return on investment
Administrative Considerations- course logistics: application, registration,
section assignment- infrastructure requirements- roles and responsibilities - faculty availability: qualifications, recruitment, hiring, orientation, training- policy adaptation/creation
Interactivity … in eLearning
Number of Students
Fully Interactive
Self-paced
Amount of interactivity
with faculty & other students
High
Low High
Program Development Process… key system elements
Analysis and Planning- need identification and assessment - congruency with mission/strategic plan- market analysis - SWOT analysis- business plan - approval process
Development and Delivery - faculty development - program and instructional design- prototype development- online course development/ production - scalable T & L technology deployment- deployment of support services - program delivery- ongoing assessment and continuous improvement
Evaluation and Assessment - learning outcomes- 5 key holistic factors
-education, access, communication, technology and satisfaction
- quality of the online experience- program effectiveness and impact- program return on investment (ROI)
Identifying a Niche … finding your spot in the eLearning marketplace
Important Factors for Individuals Interest in Taking Online Courses
• 98% - transferability
• 97% - online access to library
• 96% - interaction with instructor
• 94% - reputation of the institution
• 94% - technical support offered
• 93% - cost
• 93% - allowed to work at own pace
Market Analysis • Adults aged 18 – 50• Possess some college• 4 states in eastern U.S.A.
Course Development Model … a scalable system
PHASE 1Planning &Development
PHASE 2Component &Module Production
PHASE 3Component &Module Evaluation
Material Development Content Formatting Course Management Tool Streaming Production ADA Compliance Audio Graphics
Learning Outcomes Design and Flow Instructional Delivery Production Values
LEARNER
Course Feedback
Course Feedback
DELIVERY
CD-ROM
INTERNET
SERVER
Distance Learning Instructional Design Online Education Course Transformation
Considerations Identify Delivery Methods Course Re-design Plan Component Design Plan Production Plan
DESIGN, DEVELOPMENTMaster Faculty and IDDL Staff
LEARNER
LEARNER
LEARNER
INSTRUCTOR
Online Course Delivery Model … a scalable system
MASTER FACULTY• Plan Course • Design Content• Maintain Content• Establish Assessment Metrics• Instructor of Record
IDDL• Programmatic Support• Course Transformation• Develop Course Media• Coordinate Delivery
DLINSTRUCTOR
DLINSTRUCTOR
DLINSTRUCTOR
DELIVERY
N
N
N
DESIGN, DEVELOPMENT, DELIVERY MANAGEMENT
Course Feedback
Course Feedback
LEARNERS
INSTRUCTOR• Monitor Student Progress • Manage Logistical Tasks • Guide, Mentor, Coach• Lecture & Provide Feedback• Help Design Learning
Scalability … Cost Model Comparison (Technology Costing Methodology – WCET)
Number of Students
$
Course-Related Costs
Classroom
Mediated
High
Low High
Student-Related costs
For more information please contact:
Dr. Tom Wilkinson – Director,
Institute for Distance and Distributed Learning
Virginia Tech
www.iddl.vt.edu
www.vto.vt.edu