An Interdisciplinary Approach to Institutional Level Evaluation of An Interdisciplinary Approach to Institutional Level Evaluation of Online Learning Environments

15th Annual Sloan-C International Conference on Online Learning | Orlando, FL | October 2009

• Introductions

• APUS Mission and Demographicsg p

• Assessment Website

• Goals of the Program Review Process

• Program Review Process, Tools, and Resources

• Use of Continuous Improvement in Business Programin Business Program

• Use of Data to Drive Decisions

• Conclusion and Dialogue

Frank McCluskey, Ph.D.Executive Vice President and Provost

Phil Ice, Ph.D.Director, Course Design, Research &

DevelopmentDevelopment

Chad Patrizi, ABDDean of Business Administration

Jennifer Stephens, Ph.D.Associate Vice-President, Dean of Assessment

Dave BecherDirector, Academic Information Analysis

To educate the nation's military and public service communities and public service communities by providing respected, relevant,affordable, and student-focusedonline programs which prepare them for service and leadership them for service and leadership in a diverse, global society.

• Founded in 1991 as American MilitaryUniversity to provide affordable and convenient distance learning to service gmembers

• In 2002, expanded reach to national security,public safety and other professionals seeking affordable education and online convenience affordable education and online convenience, establishing American Public University System with member institutions: American Military University and American Public University

• We are a 100% online for-profit school that serves 53,000+ students who study online from 100 countries.

External  ReviewExternal  Review

Data Collection

Data Collection

AnalysisAnalysisFollow‐UpFollow‐Up

Program Review Meeting

Program Review Meeting

Three Year Plan

Three Year Plan

Electronic Repository of Information

Library and Learning Resources

Curriculum Assessment

External Reviewer Feedback

• Course books

• Electronic resources

• Learning strategies

• Student learning outcomes

• Instructional strategies

• Evaluation procedures

• Expert reviewer report

• Industry Advisory Council report

• Evaluation procedures

• Academic rigor

Electronic Repository of Information

A l i f f lt

Faculty

• Student demographic

Students

C i l M i

Learning Outcomes Assessment

• Analysis of facultycredentials andexpertise to ensurebreadth and diversity

• Student demographicinformation

• Enrollment History

• Growth trends

• Curricular Mapping

• Assessment measures

• Fact book

Electronic Repository of Information

Program Benchmarking

Program Directory Summary

Program Review Findings

• Program benchmarkingwith similar programsand institutions

• Evaluation of findings

• Programrecommendations

• Three year proposed

• Dean’s observations

• Meeting minutes

• Three year proposedstrategic plan

D a v e – insert slidesDave  insert slides

• Introduction

• Bachelor of Business AdministrationProgram (BBA) vs. Institutional EOC (end of course survey)

• Without Micro and Macro Economics• Without Micro and Macro Economics

• Fact Book

4.60

BBA Program vs Institutional EOC Data 2006

4.40

4.00

4.20

3 60

3.80

3.40

3.60

Clear course objectives Course assignments supported course objectives

Course materials relevant to objectives

Course assignments fairly evaluated work

Instructor's teaching style appropriate Knowledgeable instructorInstructor provided feedback to inquiries within 72 hours

Instructor provided grades and appropriate feedback within one week

Interactions between the instructor and the students were positive

Interaction among classmates created a sense of community

Electronic classroom easy to navigateAssignments easy to post Technical support questions answered within 2 days

BBA Program Average (n=1139) 4.42 4.42 4.50 4.31 4.27 4.45 4.35 4.29 4.40 4.01 4.53 4.49 3.87

Institutional Average (n=28344) 4.44 4.45 4.46 4.37 4.35 4.44 4.38 4.27 4.44 4.02 4.39 4.48 3.83

4.50

BBA Program vs Institutional EOC Data 2006 w/o Micro and Macro Economics

3.50

4.00

2.00

2.50

3.00

0.50

1.00

1.50

0.00

0.50

BBA Pro g ram Averag e (n=113 9 ) 0 .0 0 0 .00 0 .0 0 4 .3 9 4 .3 8 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0

Ins t itut io nal Averag e (n=28 3 4 4 ) 4 .4 4 4 .4 5 4 .4 6 4 .3 7 4 .3 5 4 .4 4 4 .3 8 4 .2 7 4 .4 4 4 .0 2 4 .3 9 4 .4 8 3 .8 3

Clear co urse o b ject ives

Co urse ass ig nments sup p orted

Course materials relevant to o b ject ives

Co urse ass ig nments

fairly evaluated

Ins tructo r's teaching s tyle ap p ro p riate

Kno wled g eab le ins tructo r

Ins tructo r p ro vid ed

feed b ack to

Instructo r p ro vided g rad es and app ro p riate

Interact ions b etween the

ins tructo r and

Interact io n amo ng

classmates

Electro nic class ro o m easy

to navigate

Ass ig nments easy to p o s t

Technical sup p ort

q ues t ions

C i f Community of Inquiry Framework

a process model of learning in online and blended educational environments

grounded in a collaborative constructivist view of higher educationview of higher education

assumes effective online learning requires the development of a community of the development of a community of learners that supports meaningful inquiry and deep learningand deep learning

social presence cognitive presence

LEARNINGLEARNINGLEARNINGLEARNING

teaching presence

Social Presence

the ability of participants in a community of inquiry to project themselves socially and emotionally -- as ‘real’ people

the degree to which participants in g p pcomputer mediated communication feel socially and emotionally connected

Cognitive Presence

the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of i i inquiry

Teaching Presence

the design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and d ti ll th hil l i educationally worthwhile learning

outcomes

Community of Inquiry Surveyy

9 social presence items (3 affective expression, 3 open communication, 3 group cohesion) p , g p )

12 cognitive presence items (3 triggering, 3 exploration 3 integration 3 resolution)exploration, 3 integration, 3 resolution)

13 teaching presence items (4 design & facilitation 6 facilitation of discourse 3 direct facilitation, 6 facilitation of discourse, 3 direct instruction)

CoI Survey Validation

tested in graduate courses at four tested in graduate courses at four institutions in the US and Canada

principal component factor analysis

three factor model predicted by CoIp yframework confirmed

Arbaugh, Cleveland-Innes, Diaz, Garrison, Arbaugh, Cleveland Innes, Diaz, Garrison, Ice, Richardson, Shea & Swan - 2008

APUS Utilization

American Public University System

100% online100% online

monthly course starts

CoI is the end of course survey

eight semesters of data collectioneight semesters of data collection

Application

comparison of at the University, School, Program, Course and Instructor levels

assessment of instructional design efficacy

t f l diff assessment of learner differences –demographic factors

assessment of integration of new technologies

Frank McCluskey, Ph.D.fmccluskey@apus.edu

Executive Vice President and Provost

Phil Ice, Ph.D.pice@apus.edu

Director, Course Design, Research & Development

Chad Patrizi, ABDcpatrizi@apus.edu

Dean of Business Administration

Jennifer Stephens, Ph.D.jstephens@apus.edu

Associate Vice-President, Dean of Assessment

Dave Becherdbecher@apus.edu

Director, Academic Information Analysis