Evaluating Online Learning: Issues and Strategies

Mark HawkesDakota State UniversityInnovations, Educating New GenerationsMarch 1, 2002

Presentation Objective

Discuss online learning evaluation approaches in graduate programs at two universities Identify criteria/indicators suitable for the evaluation of online learning environments

Distance Learning Literature

Evaluation . . . . . . . No Yes(79%)

(21%)

Focus . . . . . . . . . Training Education

Impact on Learning . . . . . . . No Yes

(87%)(13%)

(58%) (42%)

Instructor Student Student Student Student Student

Internet

Login Interface

Content

Resources

AssessmentCommunication

Modes

Support Services System Resources

Managementand Organizational

Information

OnlineLearning Architecture

Institutional

Familiar Online Learning Evaluation Targets . . .

interface designinstructional designstudent satisfactiontechnology accessfaculty satisfactioneconomic viabilitydepartmental capacityinterdepartmental collaboration

Common Dimensions of Effective Online Learning:

Relevant and challenging assignments Providing adequate and timely feedback through teacher-student interactionFlexibility in teaching and learning Constructing coordinated learning environments Constructing rich environments for student to student interaction

DSU’s Educational Technology Program

Students:36 Credit hour MS program80% Education (K12, Technical, Higher Ed.)20% Business/industry90% Online; 10% On campusFemale 68%; Male 32%Project-based curriculum

DSU’s ET EnvironmentPervasive technological culture Consistency between program goals and the state/region-wide initiatives Campus-wide faculty support Institutional experience in Web-based instruction delivery Multi-delivery methodsClient: teachers, teacher developers, instructors, trainers, technology coordinators, etc.Predominantly web-based delivery

An Evaluation Model . . .

IlluminativeOperation of Components And Subcomponents

IntegrativeHolistic perspective onThe learning experience

Course &Program Design

Components

Infra-structure/System

Work Flow

Interaction Impact

Process Product

Observing and Detecting Focused on Performance Functional Problems Outcomes

Infrastructure/System

Input/output devices Network speed and connectivity Network design/Topology Technical support systems and maintenance

Course and Program Design

Nature of the Design Situation Based Role of State and National Standards Sequencing/Instructional Strategies Assessment Motivation: Learning vs. Performance

Visualization Tools and MediaUser InterfaceCourse Management

Work Flow

Use of discussion toolsSoftware usageMessage redundancy (audio, video, web pages, emails).Progression Do learners progress through their

work tasks in a linear fashion? (novice-like)

Nonlinear opportunistic fashion (expert-like)

InteractionSocial and instructionalMust account for all of the following relationships:

Instructor Learner

Learners

Content

Learners

Technology

Content

Technology

Learner

Instructor

Content Assessment

Learning Resources(Technology)

LearnerRecords

Catalog info

Query Preferences

Performance

Interaction Example

(Metcalf, Snitzer, Austin, 2001)

Learner

Online Course InteractionAnnouncementsEmailDiscussion Board (WebBoard Conferencing)Synchronous text chatStreaming media using Real PlayerDesktop VideoFile LoadingOnline assessmentAudio/video clipsAudio-narrated PowerPoint'sRoom-based Video

ImpactCourse performanceCollaborative learningRetention/attrition (course and program)Professional relevance and utilityLearner productivity

Evaluation Attributes

Multi-sourced data (students, server log files, etc) Internal and external Performance based Comparison and criterion based

Student Course Ratings

3.16

3.26

3.27

3.29

3.32

3.32

3.42

3.45

3.45

1 2 3 4

1

1=Strongly Disagree; 4=Strongly Agree

The instructor wasaccessible

The instructor wasresponsive

Course was relevant todegree

Communicationtechnologies were effective

Course activities wereapplicable

Ample opportunity fordiscussion

Course format appropriatefor content

I enjoyed the course

Confident in conductingdesign

Helpfulness of Technology

3.13

3.19

3.52

3.58

3.61

3.74

1 2 3 4

1

1=Not at all helpful; 4=Very helpful

Personal communicationwith classmates orinstructor

Informational course emails

Electronic chat sessionswith instructor

Electronic chat sessionswith classmates

Course readings on website

Course texts

The breadth of this course was:

1 2 3 4 5 6 7 Not nearly enough The right amount Way too much

Compared to a traditional course

1 2 3 4 5 6 7A much narrower range About the same range A much wider range ofof material was covered of material was covered material was covered

Online: 4.61Compared to traditional 4.65 n=32

The depth of this course was:

1 2 3 4 5 6 7 Not nearly enough The right amount Way too much

Compared to a traditional course

1 2 3 4 5 6 7Material was covered in Material was covered in Material was covered in much less depth about the same depth much more depth

Online: 4.48Compared to traditional 4.42 n=32

The extent of critical thinking required:

1 2 3 4 5 6 7 Not nearly enough The right amount Way too much

Compared to a traditional course

1 2 3 4 5 6 7 Much less About the same Much more

Online: 4.61Compared to traditional 4.94 n=32

The amount of effort put into the course:

1 2 3 4 5 6 7 Much less About the same Much more

Compared to a traditional course

1 2 3 4 5 6 7 Much less About the same Much more

Online: 5.65Compared to traditional: 5.26 n=32

Evaluation Processes

Mostly formativeMixed data collection and analysis methods (document analysis, student artifact analysis, survey, interview, text analysis, etc.)Course and program evaluation

Typical Problems with Online Courses

Facilitating and encouraging collaboration Time managementStudent proficiency with course toolsAmbiguous directionsTimeliness of feedback

Factors Beyond ID Control

Student sophistication with technology tools System capacity Learner availability/accessibility Enthusiastic, responsive instructorGood learner supportMotivated learners

How to Design and Effective Online Course?

Follow basic ID principalsBuild a climate of disclosure and full participation Institute informal student evaluation and check-in mechanisms Active and intensive instructor participation Build in as much interactivity as possibleCreate visually interesting screens/pagesEnsure instructions are very clearMulti-mode interaction is criticalRedundancy