David Ackerman, Associate VP Crystal Butler, Research Associate.

David Ackerman, Associate VPCrystal Butler, Research Associate

1 – MAKE IT COUNT

2 – MAIN METHOD

3 – GETTING RESULTSwhat did we find out?

4 – LESSONS LEARNED...and what does it mean?

AGENDA

SAKAI MARKET SHAREMAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

what determines market share?

how do we get good data?

GETTINGRESULTS

1 – COUNT METHOD

2 – RESPONSE RATE

one institution, one vote: enrollment rates are not reflected

since enrollments vary, sample data can't be generalized

commercial LMS providers won't share client lists; open source registries are poorly managed

THE ISSUES

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

GETTINGRESULTS

3 – ABSENT PROVIDER DATA

SAKAI MARKET SHARE

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Counting installations rather than users could present a skewed perspective of the changing LMS landscape.

COURSEMENU

MAKE ITCOUNT

THE COUNT

GETTINGRESULTS SAKAI MARKET SHARE

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

LMS market share as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, including Community Colleges.

COURSEMENU

MAKE ITCOUNT

THE COUNT


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

LMS market share as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, with Community Colleges removed.

COURSEMENU

MAKE ITCOUNT

THE COUNT


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Four-year+ schools had an average response rate of 28.4%....making it impossible to generalize about student usage across colleges and universities with widely varying enrollment rates.

COURSEMENU

MAKE ITCOUNT

RESPONDENTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Commercial vendors do not share their complete customer lists.

COURSEMENU

MAKE ITCOUNT

LMS PROVIDERS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Open source providers may try to track system installations...

COURSEMENU

MAKE ITCOUNT

LMS PROVIDERS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

...but have incomplete control over implementation and client downloads, so cannot provide accurate usage statistics.

COURSEMENU

MAKE ITCOUNT

LMS PROVIDERS

SAKAI MARKET SHARE

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

As the end users, learners should be tallied to determine market share. By completion of their fourth year in college, the ratio of students who will have used a school's primary LMS approaches 100%.

COURSEMENU

MAKE ITCOUNT

THE USERS


1 – IDENTIFY INSTITUTIONS

2 – DETERMINE A DATA GATHERING METHOD

select schools and get enrollment numbers

how to count 100% of the student population of interest?

3 – ESTABLISH A PROCESS the plan, pitfalls, and (striving for) perfection

THE STEPS

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

MAINMETHOD


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Schools were selected using the U.S. Department of Education's Institute of Education Sciences database...

COURSEMENU

MAKE ITCOUNT

IPEDS

GETTINGRESULTS

MAINMETHOD SAKAI MARKET SHARE

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

with the following criteria:• 4-year+ not-

for-profit• degree

granting• based in the

U.S.• 1,000 or more

students• have a physical

campus

COURSEMENU

MAKE ITCOUNT

IPEDS

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

We have a lot of work to do: how to determine the LMS in use by over 1,500 schools?

COURSEMENU

MAKE ITCOUNT

IPEDS RESULTS

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Web search.

With a large data set requiring repetitive search operations and intelligent evaluation of results, we had an ideal use case for Amazon's Mechanical Turk.

COURSEMENU

MAKE ITCOUNT

MECHANICAL TURK

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MTurk works best when the instructions are standardized, so we had to establish a web search that would return optimal results. We tested the term 'learning management system' and related terms suggested by Google and other sources...

COURSEMENU

MAKE ITCOUNT

THE SEARCH

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

...but querying specific LMS names returned the best links.

COURSEMENU

MAKE ITCOUNT

THE SEARCH

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

COURSEMENU

MAKE ITCOUNT

THE SEARCH

GETTINGRESULTS

MAINMETHOD

1 – MTurk TESTING

2 – WORKER QUALIFICATION

• first run: 100 items, 2 Master Classification Workers each (200 HITs)

• second run: 50 items, 3 Master Workers each (150 HITs)• third run: 400 items, 3 Master Workers each (1200 HITs)

• goal: create workforce with 95% accuracy or greater

• reality: standards had to be set at 85% accuracy to generate a large enough worker base

SAKAI MARKET SHARE

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

COURSEMENU

MAKE ITCOUNT

GETTINGRESULTS

MAINMETHOD

• use 3 workers per item; verify LMS choice for all worker disagreements

• do 5% audit on all worker agreements• compare final LMS list to provider information found on

web

4 – DATA VERIFICATION

• 3 Custom Qualification Workers per item• remaining institution list split in half and run sequentially in

two sets

3 – DATA COLLECTION

THE SEARCH

SAKAI MARKET SHARE

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Mechanical Turk provides analytics for completion velocity, average hourly pay rate, and percent HITs submitted.

COURSEMENU

MAKE ITCOUNT

MTurk TESTING

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Of the 49 workers who attempted our HITs, only 10 performed well enough to meet our qualification standards.

COURSEMENU

MAKE ITCOUNT

WORKER QUALIFICATION

GETTINGRESULTS


MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

Audits consistently found a 95% accuracy rate for worker agreements. • Workers:

.95 x 3621 = 3440• In-house

search:1.0 x 1263 = 1263• Accuracy:

4703/4884 = 96.29%

We are confident that our results are not off more than +/- 5%

COURSEMENU

MAKE ITCOUNT

DATA VERIFICATION

GETTINGRESULTS


1 – OUR FINDINGS

2 – A COMPARISON

LMS market share based on 2010-2011 1-year headcount

how do our results differ from the CCS count?

3 – THE SPREADSHEET our data by LMS

THE RESULTS

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

MAINMETHOD


THE NUMBERS

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

MAINMETHOD

GETTINGRESULTS

Based on a one-year enrollment of 13,723,024 students.

SAKAI MARKET SHARE

THE COMPARISON

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

MAINMETHOD

GETTINGRESULTS

Campus Computing Survey Count by

Institution

New York University Count by Enrollment

SAKAI MARKET SHARE

ENROLLMENTS

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

MAINMETHOD


OUR DATA

MAKE ITCOUNT

MAINMETHOD

LESSONSLEARNED

MAKE ITCOUNT

MAINMETHOD


https://docs.google.com/a/nyu.edu/spreadsheet/ccc?key=0At_Bu9HtBEIldHdZa3pSUzZUQldfOFZlSVRKSGZHVHc%23gid=0

1 – GROWTH OF CANVAS

2 – SAKAI

6.2% market share since founding in 2008, with many recent migrations

8.3% market share since deployment in 2005, with few recent migrations and many previous Sakai installations having been replaced by other systems

3 – OPEN SOURCE 26.9% of primary LMSs by enrollment are open source, less than the 33.6% reported by the CCS count; Blackboard is often the primary LMS at schools with smaller parallel or test installations of other systems

CONCLUSIONS

MAKE ITCOUNT

MAINMETHOD

MAKE ITCOUNT

MAINMETHOD

GETTINGRESULTS

LESSONSLEARNED SAKAI MARKET SHARE

1 – PUBLISH PAPER

2 – FIND HOME FOR SPREADSHEET OF DATA

Include analysis as to statistical significance of comparison survey

This will allow schools to correct any errors as well as report a change in LMS

3 – SHARE DATA

FUTURE WORK

MAKE ITCOUNT

MAINMETHOD

MAKE ITCOUNT

MAINMETHOD

GETTINGRESULTS


QUESTIONS

MAKE ITCOUNT

MAINMETHOD

MAKE ITCOUNT

MAINMETHOD

GETTINGRESULTS


David Ackerman, Associate VP Crystal Butler, Research Associate.

Documents

Transcript of David Ackerman, Associate VP Crystal Butler, Research Associate.