Logging On To Improve AchievementEvaluating the relationship between use of the Learning Management System, student characteristics, and academic achievement in a hybrid large enrollment undergraduate course

Research Highlights: Presentation to SoLAR Storm

November 15, 2012John C Whitmer (jwhitmer@calstate.edu)

Committee Chair: Dr. Paul Porter, Sonoma State University

Slides: http://slidesha.re/sFKjcm

Introduction

• Educational Doctorate Degree (EdD) candidate (University of California Davis & Sonoma State University)

• Advanced to candidacy, defending ~ January 14

• Associate Director, California State University LMSS Project, Chancellor’s Office

Me

Presentation Outline

1. Study Case & Context

2. Results for Instructional Practices

3. Results for LMS Data Analysis

4. Conclusions

STUDY CASE & CONTEXT

Problem: Student Graduation• Less than 50% of college/university students graduate

within 6 years• California State University: 52.4%

(first-time freshman, 2000 cohort) (CSU Analytic Studies, 2011)

• Students from under-represented minority racial/ethnic groups graduate at much lower rates • California State University: 38.3%

(African American students, first-time freshman, 2000 cohort) (CSU Analytic Studies, 2011)

• Contributing factor: mega-enrollment intro courses• Infrequent interaction, prevent faculty/student relationships

Case: Introduction to Religious Studies

• Redesigned to hybrid delivery through Academy eLearning

• Highest LMS usage entire campus Fall 2010 (>250k hits)

• 373 students (54% increase)

• Bimodal results• 10% increased SLO mastery• 7-11% increase in DWF

54 F’s

Research Questions

1) Is there a relationship between student LMS usage and academic performance? Does this relationship vary by the pedagogical purpose underlying LMS usage? (correlation)  

2) Is there a relationship between student background characteristics or current enrollment information and academic performance? (correlation)

3) Does analyzing combined student characteristics and current enrollment information increase the predictive relationship between combined LMS usage data and student success? (multivariate regression)

4) Does a student’s economic status and student of color status vary the predictive relationship between combined LMS usage, combined background characteristics and current enrollment information? (multivariate regression, restricted model)

Independent Variables: Student Characteristics

Independent Variables: LMS Usage

Research Methods (Cliff’s notes version)

1. Extract data, validate with appropriate “owner”

2. Transform variables • measures of interest (e.g. “URM”, not race/ethnicity)• analysis methods (categorical into numeric)

3. Examine data for • outliers, missing data, data distributions, etc.• colinearity between variables (e.g. independence)

4. Join data into single data file, collapse to one record/student

5. Run analysis

Results for Instructional Practices

Correlation: LMS Usage w/Final Grade

Scatterplot of Assessment Activity Hits v.

Course Grade

Correlation: Student Char. w/Final Grade

Most interesting finding (so far):

Smallest LMS Use Variable

(Administrative Activities)

r=0.3459

Largest Student Characteristic

(HS GPA)

r=0.3055>

Regression R2 Results Comparison

RESULTS FOR LMS DATA ANALYSIS

Lms Logfiles: “Data Exhaust”

1. Logfile tracks server actions (not educationally relevant activity)

2. Duplicate logfile hits for single student action

3. To remedy, filtered logfiles by:• Time (> 5sec, <3600 sec)• Actions (no “index views”, more)

Logfile Data Filtering Results

Discus

sion

Activi

ty H

its

Conte

nt A

ctivi

ty H

its

Asses

smen

t Act

ivity

Hits

Mai

l Act

ivity

Hits

Admin

istra

tive

Activi

ty ..

.0

50

100

150

200

250

300

350

400

450

382

151

58 4926

54 5123 36

16

Final data set: 72,000 records (from 250K+)

LMS Use Consistent across Categories

Factor Analysis of LMS Use Categories

Missing Data On Critical Indicators

Conclusions

1. At the course level, LMS use better predictor of academic achievement than any student characteristic variable. Behavioral data appears to supercede demographic information (what do, not who are).

2. Moderate strength magnitude of complete model demonstrates relevance of data, but suggests that refinement of methods could produce stronger results.

3. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.

Ideas & Feedback

Potential for improved LMS analysis methods:• social learning • activity patterns • discourse content analysis• time series analysis

Group students by broader identity, with unique variables:• Continuing student (Current college GPA, URM, etc.• First-time freshman (HS GPA, SAT/Act, etc)

Contact Info

John Whitmer

jwhitmer@calstate.edu

Skype: john.whitmer

USA Phone: 530.554.1528

By WingedWolfDamián Navas