Introducing the indicators project

The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors

Colin BeerDavid JonesKen Clarkhttp://tr.im/FvRQ

Overview

http://www.flickr.com/photos/atbaker/1577665185/

Context

Feature usage

Participation, grades

and other factors

Future work and Reflections

http://www.flickr.com/photos/dnorman/251646154/

The data that we, educators gatherand utilize is all but garbage.(Wiley,

2009)

http://www.flickr.com/photos/riverap1/3875718119/

almost every university is planning to make use of an LMS (Salmon,

2005)

They are becoming ubiquitous at

universities around the world(Coates et al, 2005)

http://www.flickr.com/photos/dsifry/2452527887/

Are sites ready for start of term?

Are students using the site?How?

With what impact?

What makes “good” LMS use?

Are thereproblems?

If I do “x” what happens?

Is it worth it?

http://www.flickr.com/photos/80417459@N00/2471314610/

What’s going on within your LMS/course site?a) Asking staff and/or

studentsb) Manual checking of LMSc) LMS reporting facilitiesd) Data mining + other analyticse) Opinion/self-reportingf) Otherg) None


Mostly ad hoc

Some surveys

Mostly LMS reporting

One manual check (Gonch,

2007)

Peer review?


Mostly ad hoc

Some surveys



2007)

No widespread view


Mostly ad hoc

Some surveys



2007)

No widespread view

Only what is in the LMS

Limited views


Mostly ad hoc

Some surveys



2007)

No widespread view


Human recollection

Limited views


Mostly ad hoc

Some surveys



2007)

No widespread view


Human recollection

Time-consuming & error prone

Limited views


Mostly ad hoc

Some surveys



2007)

No widespread view


Human recollection


No lognitudinal examination

Limited views


Mostly ad hoc

Some surveys



2007)

No widespread view


Human recollection


No lognitudinal examinationNo cross LMS comparison

Limited views


Mostly ad hoc

Some surveys



2007)

No widespread view


Human recollection


No lognitudinal examinationNo cross LMS comparison

No cross institutional comparison

Limited views


“LMS”

Webfuse – 1997-2009

Pre-2010

2010-

WebCT – 1999-2003/4Blackboard – 2004-2009

Moodle

http://www.flickr.com/photos/anders-vindegg/3408838186/

We have lots of data

Examining it might reveal interesting patterns

Useful for L&T practice, management and research

http://indicatorsproject.wordpress.com

Enabling comparisons of LMS usageacross institutions, platforms and

time

http://www.flickr.com/photos/david_jones/36379215/


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data

Useful Data/Informat

ion


ion


ion


ion

T&L Practice Research


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling

What


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling

What

Why


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


Enabling

What

WhyHow

Overview


Context

Feature usage


and other factors

Future work

http://www.flickr.com/photos/chokingsun/3495110670/

There are more similarities than differences among LMS (Black et al,

2007)


TransmittingContent

CreatingClass

Interactions

ComputerBased

Instruction

Move to features in unexplored categories, until each category is considered for different learning needs

Level 1: Most used Level 2: Moderately used Level 3: Rarely used

• Content/files• Announcements• Gradebook

More than 50% of courses More than 20% of courses (much) less than 20% of courses

EvaluatingStudents• Quizzes• Drop box

Evaluatingcourse and instructors

• Surveys• Chat• Discussion forum• Email

• Some quizzes• Adaptive release

(Malikowski et al, 2007)


What we did

CQU

Wf BbLMS logs


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage


Malikowskitop & bottomrange


Blackboard


Webfuse

http://www.flickr.com/photos/wadem/2730257498/


The particular trajectory of emergence is not wholly

determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two(Jones,

1999)

http://www.flickr.com/photos/nedrichards/3234490934/

Future Work

Will Moodle be different?

Why?

What about other institutions?

Are Malikowski ranges out of date?

When is a feature adopted?

How

…….

Overview


Context

Feature usage


and other factors

Future work and Reflections

http://www.flickr.com/photos/wolfgangstaudt/2279651479/

The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course (Dawson et al,

2008)

Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance. (Dawson, 2009; Morris et al,

2005)


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage

Student records


CQU

Wf BbLMS logs

LMS Independent

Data

What we did


Feature usage

Participation, grades,external factors

Student records


Hits on course site

Hits on course forum

Distance education students


CQ on-campus students


AIC students


Distance education students

# of replies

# of posts


CQ on-campus students


AIC students



2008)


2005)


Mixed design ANOVA

All reported effects are significant

Significant interaction effectbetween grades and participation


AIC = emphasis on face-to-face

NESB?

FLEX/DE = no face-to-face

CQ = Quite a bit emphasis face-to-face

Course site designedby CQ-based staff(usually)

Taxonomy of critical success factorsfor quality web-supported learning(Fresen,

2007)

Lecturer interaction/facilitationAcademic background of lecturer/instructional design

http://www.flickr.com/photos/orinrobertjohn/3321067724/


Course groups based on staff hits

Group Courses Student-courses

Students/course

Definition

High 678(17%)

156,770 231 x>3000 hits

Medium 980(25%)

143,314 146 3000>x>1000

Low 1474(37%)

102,214 69 1000>x>100

Super low 849(21%)

68,354 80 100>x>0


Hits

High staff participation courses


Replies & posts

High staff participation courses


Hits

Super low staff participation courses


Replies & posts

Super low staff participation courses


High

Posts comparisons

Superlow



2008)


2005)


Future Work

- Course size

Examination of super-low

Networks of interaction (Bakharia et al, 2009; Dawson,

2009)

Forum content

Taxonomy of critical success factorsfor quality web-supported learning(Fresen,

2007)

Lecturer interaction/facilitationAcademic background of lecturer

and instructional design



Course offerings with

Staff with grad certs in L&TSignificant input from instructional designerStaff who have received teaching awards

Staff teaching education coursesNone of the above

n=167

n=4

n=196

n=3288

N=362


None of the above


Grad cert in L&T


Teaching awards


Education discipline


Instructional design

Overview


Context

Feature usage


and other factors

Future work & Reflections

http://www.flickr.com/photos/mikebaird/2985066755/

http://www.flickr.com/photos/david_jones/36379215/

http://www.flickr.com/photos/dmatos/540688952/

http://www.flickr.com/photos/mwichary/2188958154/

http://www.flickr.com/photos/sylvar/1138341328/

http://www.flickr.com/photos/fncll/145149313/


Institution A

LMS 1 LMS 2

Institution B

LMS

Institution C

LMS

LMS Independent

Data


ion


ion


ion


ion


What

WhyHow

EnablingLittle

bit

Little bit

Little bit Not

much

http://www.flickr.com/photos/fncll/145149313/

http://www.flickr.com/photos/jamuraa/813966437/

http://tr.im/FvRQ

http://indicatorsproject.wordpress.com

Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends 51(2): 35-39.

Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.

Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.

Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230

Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning 6(3): 351-362.

Gonch, B. (2007). An inventory of Term 2 2006 online courses at CQU. Rockhampton, Central Queensland University: 18.

Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.

Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research 36(2): 149-173.

Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education 8(3): 221-231.

Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology 13(3): 201-218.

Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.

Introducing the indicators project

Education

Transcript of Introducing the indicators project