Using Large-Scale LMS Data Portal Data to Improve Teaching and Learning (at K-State)

USING LARGE-SCALE LMS DATA PORTAL DATA TO IMPROVE TEACHING AND LEARNING (AT K-STATE)

INTERNATIONAL SYMPOSIUM ON INNOVATIVE TEACHING AND LEARNING AND ITS

APPLICATION TO DIFFERENT DISCIPLINES

DIGITAL POSTER SESSION

SEPT. 26 – 27, 2017

KANSAS STATE UNIVERSITY – TEACHING & LEARNING CENTER

SESSION DESCRIPTION

• With any learning management system, a byproduct of its function is data, which may be

analyzed to improve awareness, decision-making, and actions. At Kansas State University, its

Canvas LMS instance recently made available its cumulative data from its first use in 2013.

These flat files open a window to how the university is harnessing its LMS, with some macro-

level insights that may suggest some areas to improve teaching and learning. This session

describes some approaches to informatizing this empirical “big data” with some basic

approaches: reviewing the data dictionary, extracting basic descriptions of the respective

data sets, conducting time-based comparisons, surfacing testable hypotheses from data

inferences, and conducting other data explorations. This introduces initial data analysis work

only, but this does not preclude front-end analysis of courses at the micro level, relational

database queries of the data, and other potential follow-on work.

2

PRESENTATION ORDER

• K-State Online Canvas LMS data

portal data

1. About Courses

2. About Course Sections

3. a. About Assignments

b. About Submitted Assignments

4. About Quizzes

5. About Discussion Boards

6. About Learner Submitted Files

7. About Uploaded Files

8. About Wikis and Wiki Pages

9. About Enrollment Role Types

3

PRESENTATION ORDER (CONT.)

10. About Groups

11. About Users and Workflow

States

12. About Course Level Grades

(based on Enrollments)

13. About Conversations (In-System

Emails)

14. About Third-Party External Tool

Activations

15. About Course User Interface (UI)

Navigation Item States

• Summary

4

K-STATE ONLINE CANVAS LMS DATA PORTAL DATA

5

K-STATE ONLINE CANVAS LMS DATA PORTAL DATA

• LMS data portal data comes from event logs and trace files captured as part of the

provision of learning management system (LMS) services

• These originate from SQL and include structured quantitative data and text data

• K-State data comes in 79 data tables, which download as .gz files in a zipped

folder

• .gz files are opened using 7Zip

• These are then files without extensions

• .csv (comma separated values format) may be added on the end to make the files readable

in Access, SQL, Excel, etc.

8

K-STATE ONLINE CANVAS LMS DATA PORTAL DATA (CONT.)

• Data is updated every day with K-State’s Canvas license.

• The data is already in digital format.

• The data structures are in classic data table format with student data in rows and

variable data in columns.

• What default settings are chosen for an LMS instance will likely have a

weighty influence on the uses of the LMS functions.

9

DATA HANDLING

• There are millions of rows of data in some of the tables.

• The data has to be handled properly so there is no lossiness in the handling.

• As always, a pristine and “untouchable” (you can’t revise or change the raw set in any way)

raw set of data should be maintained as a hedge against data mishandling. A working set of

this data can be accessed for data cleaning and queries…

• The working set of data requires a little cleaning to be useful.

• Outlier data may have to be eliminated so as not to skew curves.

• Data garble should be omitted. Sometimes, garble is collected. Sometimes, people use the

LMS in a way that create garble.

10

A DATA DICTIONARY OR “SCHEMA DOCS”

• There is a data dictionary or “Schema Docs” that describe the unlabeled

column data. Not all variables mentioned in the data dictionary are used

because of different choices for different instances (what universities choose to

collect or not collect, use or not use)…and because of changes in data

management over time…and other factors. (Some data columns are

discontinued / “deprecated” / no longer supported.)

• Many of the categories of data are instantiated in multiple data tables, so these have to

be combined (union-joined) for queries involving the full set of N.

11

THIS DATA ANALYSIS APPROACH

• The available information and how it is queried can elicit insights for

awareness and decision-making.

• This approach here shows uses of computers to capture meaning. This does not

preclude some “human close reading,” but the amounts of data require

computers to some degree.

12

ADDITIONAL ANALYTICAL APPROACHES

• Data analysis never happens in a vacuum.

• The “owner” (“brand ambassador”) of the instance is a good source of information.

• The back-end information from the LMS data portal can be combined with

front-end accesses for deeper insights about how the LMS can be used.

• The digital data may be compared with other data from other sources

assuming unique identifier columns may be identified and used. (These include

primary and foreign key columns.)

13

ADDITIONAL ANALYTICAL APPROACHES (CONT.)

• If baseline data is available from other comparable and non-comparable

instances, those could be informative.

• Are there “proper” or normative levels of activities that should be observed for LMSes in

particular stages of a life cycle?

14

ANALYTICS APPROACHES

Given the use of flat files and without direct

application of primary and foreign keys…

Given database queries…

Given computation-based linguistic analysis tools…

Given qualitative data analysis tools…

What is knowable from the data?

What would benefit teaching and learning (T&L)?

15

ANALYTICS APPROACHES (CONT.)

DESCRIPTIVE

• How is the LMS instance being used

by the faculty, staff, and

administration at the university?

• How has the LMS instance been used

over time?

PRESCRIPTIVE

• What are ways to improve the

university’s uses of the LMS and the

related integrated tools?

• How can the LMS be used in a

beneficial way into the future?

16

PRESENTATION:

• 15 areas of insights in the Canvas LMS (with Oct. 2016 data) expressed as

data visualizations

• Ways to interpret the available data (in terms of macro-level use of the LMS)

• Ways to harness that data for improving online teaching and learning with

LMS data and beyond

17

1. ABOUT COURSES

18

COURSE VISIBILITY AND T&L

• In the lead-up period to an academic term, are courses publicly viewable to

learners who may want to acclimate and get a start on the work?

• Follow-on questions:

• If so, what is available, and what can learners see and do?

20

COURSE WORKFLOW STATES AND T&L

• How many courses are “hard concluded” (completed) at a particular time of

the term (vs. how many should be)?

• “Claimed” courses are undeleted ones that have not yet been published, and

how many are in this state (which requires helpdesk or higher level support).


• Are there ways to head off accidental deletions of courses?

22

2. ABOUT COURSE SECTIONS

23

LIFE CYCLE STATE FOR COURSE SECTION AND T&L

• How many courses on the system are active vs. deleted?


• What is a healthy balance of active to deleted courses? Why?

• Online courses can be wholly recreated from ground-up without deletion…but deletion itself is also

low-cost. It might signal that a course may not be used again. If so, why?

• Sometimes, people delete courses because they don’t want the course to show up on their course

listings. How should that be handled instead (using the stars to select which courses initially show up

on the user dashboard).

• Sometimes, people delete courses willy-nilly, and that requires reinstating a deleted course.

25

DATE RESTRICTION ACCESSES FOR COURSE SECTIONS AND T&L

• What are the proportion of course sections in the LMS instance that have

restricted enrollments to section availability dates?


• When is there the “use case” of restricted section access to defined dates? Is that the

best way to handle that “use case” / those “use cases”? Are there negative unintended

consequences of using such features or not using such features?

27

ABILITY TO SELF-ENROLL IN A SECTION OR NOT AND T&L

• There are two general types of sections when it comes to online courses. One

is a general section linked to a formal course. These are created based on

learner enrollments in an online enrollment system.

• Others are sections created by instructors to enable segmentation of courses

for different assignments or tracks.

• Manual self-enrollment to a section applies to instructor-created sections with opt-in

learning tracks. Assignment to sections also apply to instructor-created sections.

29

ABILITY TO SELF-ENROLL IN A SECTION OR NOT AND T&L (CONT.)


• How important is it for learners to be able to select their own sections in respective

learning sequences?

• What are the methods that instructors use to assign learners to sections (can either be

random or manually)?

30

3A. ABOUT ASSIGNMENTS

31

TYPES OF ASSIGNMENTS AND T&L

• Is there a baseline count for assignment types? Are there optimal mixes for a

university of K-State’s size?

• What does the frequency of the various assignment types mean for how the

assignments are being used at the university?

• What is the proportion of graded to ungraded assignments, and which are

preferable when?

• Are ungraded assignments used for formative assessments to support learning, and are

these opt-in or required? (and if so, in what contexts?)

33

TYPES OF ASSIGNMENTS AND T&L (CONT.)


• How can a university’s administration and staff encourage more media creation

assignment types (if relevant to the learning)?

34

TIME FEATURES FOR ASSIGNMENTS AND T&L

• What types of assignments fit into each category of time features?

• Why are so many assignments without time deadlines or allotments?

• Is this a negative in an LMS that has an auto-created calendar and auto-created syllabus

based in part on deadlines?

• How do instructors use deadlines on assignments? And the converse: How do instructors

not use deadlines on assignments?

• Is it beneficial to learners to have no deadlines or some deadlines or all-

defined deadlines? And for which assignments?

36

TIME FEATURES FOR ASSIGNMENTS AND T&L (CONT.)


• How soon before an assignment is due is it unlocked? Would learners do better if they

had more time to prepare for an assignment before its unlocking?

37

MAIN THEMES AUTO-IDENTIFIED IN ASSIGNMENT NAMES AND T&L

• In terms of assignment names, what are the most frequent words used?

• What do these (particularly subtopics) suggest about the work that learners do?


• What are some “long tail” terms from this set of assignment names? The terms in the

“long tail” are those used only infrequently.

• What about controversial terms used in the assignment names?

39

SOME LINGUISTIC FEATURES OF THE ASSIGNMENT TITLES AND DESCRIPTIONS AND T&L

• Assignment titles and descriptors rank very high on analytic features (92nd

percentile). They rank in the 73rd percentile in clout and 65th percentile on

tone, but on the 12th percentile for authentic tone (warmth).


• Are there ways the improve the sense of human warmth in assignment descriptions?

Would that be beneficial or harmful for the learning?

41

DELVING INTO TOPICS OF INTEREST AND T&L

• It is possible to select terms and phrases (unigrams / one-grams, bigrams /

two-grams, three-grams, four-grams, etc.) to explore in the text set, to see the

words leading up to the target terms and the terms leading away. In the

prior word tree, “lab” was the target term.


• In assignment titles and assignment descriptions, any number of terms may be of interest.

43

GRADES VIEWABLE BY STUDENTS? MUTED VS. UNMUTED ASSIGNMENTS AND T&L

• A minority of student assignments’ grades are muted (whether temporarily or

permanently) in a course. Also, in some courses, some or all of the grades

may be muted.

• What are some ways that instructors use grade muting?

• What does grade muting enable bureaucratically or otherwise (such as for grade

adjustments)?

• How may grade muting enable learners to learn with less self-imposed or other imposed

pressure?

45

GRADES VIEWABLE BY STUDENTS? MUTED VS. UNMUTED ASSIGNMENTS AND T&L (CONT.)


• How aware are instructors of the assignment / quiz grade muting and unmuting functions?

What about learners?

• Do learners miss out on the benefits of assigned grade and other feedback if assignment

muting is applied?

• Do learners miss out on the benefits of not seeing assigned grades, such as less pressure

and less anxiety, if grade muting is not applied?

46

ASSIGNMENT WORKFLOW STATES AND T&L

• Why are assignments put into published, unpublished, and deleted states?


• Are there ways to improve the teaching and learning experience for learners by making

assignments more readily available in a sequence such as in modules?

• Are there ways to improve the learning experience by rolling out learning in time, so

learners are not overwhelmed by an entire revealed course at the beginning?

48

SURVIVAL FUNCTION OF ASSIGNMENTS TO UPDATE AND T&L

• How long do assignments “survive” before they are updated?

• What does it mean that some assignments may not have been updated for over a

thousand days (a little less than three years)?

• What does it mean that assignments that are updated tend to be updated

shortly after they were created? Does this mean better quality of update or

not?

50

SURVIVAL FUNCTION OF ASSIGNMENTS TO UPDATE AND T&L


• How quickly are assignments used after they are created? How often are assignments

updated as soon as there is feedback from learners?

• How many assignments are never updated after the first point-of-creation? What are

the proportions?

• Why do instructors and GTAs update assignments?

• What are the most common updates to assignments? Are these improvements beneficial

to the learning or not?

51

3B. ABOUT SUBMITTED ASSIGNMENTS

52

GRADES SUBMITTAL COUNTS FOR COMPLETED ASSIGNMENTS AND T&L

• A majority of the submitted assignments have received grades, but a not-

unsizable amount have not.


• How quickly do learners expect grades to arrive? What actions do they take if grades

haven’t arrived within a certain amount of time?

• How often do they check their grades?

• How much weight do learners give to the grades they receive?

54

4. ABOUT QUIZZES

55

A SURVEY OF QUIZ TYPES AND T&L

• In terms of quiz types, a majority are “assignment” types, then practice

quizzes, then graded surveys, and then surveys. What are the respective

functionalities of each? How many at a campus are aware of the various

types of quizzes and their respective functionalities?


• How do the uses of the LMS instantiate the various types of quizzes? What are some

constructive models for the respective uses of each?

57

QUIZ QUESTION TYPES IN THE LMS INSTANCE AND T&L

• A majority of question types used in the LMS are based on automated

assessment. Some—essay questions, file uploads, short answer questions, text

only questions—often require human interventions.


• What sorts of assignments use human-intervention-type questions?

• Is there higher quality feedback with non-automated types of assessments by experts (vs.

GTAs)?

59

QUIZ QUESTION WORKFLOW STATES AND T&L

• A majority of quiz questions are unpublished. Some are published. A small

amount are deleted.


• What quiz questions are deleted, and why? Are these replaced?

• How many of the quiz questions are from third-party content providers?

61

AN INCLUSIVE SCATTERPLOT OF QUIZ POINT VALUES AND T&L

• The min-max range on quiz point values is 0 – 23,000.

• A majority of quiz values are very low comparatively.

• The average point value of a quiz in the LMS is 33 points (without zeroes

integrated) and 28 points (with zeroes integrated).

63

AN INCLUSIVE SCATTERPLOT OF QUIZ POINT VALUES AND T&L (CONT.)


• What do you have to do to pass a 23,000 point quiz?!

• If defined, how much time is allowed / expected for a quiz?

• What does a low-value assessment look like? A high-value assessment?

• What are typical quiz designs? What are atypical quiz designs?

64

HISTOGRAM OF QUIZ POINT VALUES IN LMS INSTANCE (WITH A NORMAL CURVE) AND T&L

65

SURVIVAL CURVE OF DELETED QUIZZES AND T&L

• Quizzes that are ultimately deleted tend not to last very long.


• Why are quizzes deleted (instead of revised)?

67

ONE MINUS SURVIVAL FUNCTION CURVE FOR DELETED QUIZZES AND T&L

• Quizzes that last a certain number of days tend to survive without being

deleted. Why?


• What is it that instructors look for in a quiz to ensure that they will continue to use it?

• Once instructors have committed to a quiz, how long will they tend to use that quiz for

without revision?

69

HAZARD FUNCTION FOR DELETED QUIZZES AND T&L

70

5. ABOUT DISCUSSION BOARDS

71

TYPES OF DISCUSSION BOARDS: ANNOUNCEMENT VS. DEFAULT AND T&L

• Discussion topic types may be either “announcement” or “default” (blank). An

“announcement” type has text in the body; a “default” one just has a title but

no body text or prompt.


• How are announcement discussion boards set up for teaching and learning? How are

default discussion boards set up for teaching and learning?

• Do learners do better with more prompts for contents or not?

73

WORKFLOW STATES OF DISCUSSION BOARDS AND T&L

• Discussion board topics may be in various states: unpublished, active, locked,

deleted, and post_delayed. When are these various states practically

applied in a live learning context? To what end?


• How are these various states of discussion boards used and instantiated in the LMS

instance? Which are beneficial to learning, and which not? Which are beneficial to

teaching, and which not?

75

ACTIVE VS. DELETED DISCUSSION BOARD ENTRIES (REPLIES) AND T&L

• A majority of discussion board entries are left active and available, but a

minority are deleted.


• Which ones end up being deleted, and why?

• What information is lost with the deletion of discussion board entries (replies)?

77

6. ABOUT LEARNER SUBMITTED FILES

78

HANDLING OF LEARNER SUBMISSIONS AND T&L

• In one slice-in-time, a majority learner-submitted assignments (file uploads)

were graded, and a lesser amount was not graded. A small minority was

auto-graded (maybe code uploads corrected with a scripted auto-grader?).

• What types of assignments do instructors choose to auto-grade?

• What types of assignments do instructors choose to self-grade / manually

grade?

80

HANDLING OF LEARNER SUBMISSIONS AND T&L (CONT.)


• What sorts of files are requested in file upload assignments? Diagrams? Maps? Photos?

Designs? Audio files? Video files? Papers? Others?

• What are some ways that learners benefit from having fewer auto-graded files? What are

some costs to using a fair amount of instructor-graded works?

• How quick of a turnaround do learners expect for learner-submitted assignments?

• How much feedback do they expect?

• How many of these are peer-assessed vs. GTA vs. self vs. instructor assessed?

• How many of these assignments are made public to the course learners, as in an online

gallery?

81

SOME COMMON WORDS FROM COMMENTS MADE ON SUBMISSIONS AND T&L

• When learners submit file uploads, they often type text into the field accompanying the file

upload. The prior word cloud sounds frequency counts of words related to comments made with

the digital file submissions.


• What are substantive content terms used in the text learners use when uploading digital files?

• How positive or negative are the sentiments expressed as learners are uploading the files?

• What are the main purposes of the textual contents when learners share a message with their instructor when

uploading a file for an assignment?

• What are common questions when learners share a message with their instructor when uploading a file for an

assignment?

83

SUBMISSION COMMENT PARTICIPATION TYPE AND T&L

• Submission comment types may be categorized into three types: admin,

author, and submitter.

• The admin may be the instructor.

• The author may be whomever wrote the message.

• The submitter may be whomever submitted the file. (The data dictionary does not seem

clear on this.)

85

SUBMISSION COMMENT PARTICIPATION TYPE AND T&L (CONT.)


• What sorts of learning interactions go on around uploaded files?

• What are potential learning gains for the instructors? The co-learners? The target learner

who shared the file?

86

7. ABOUT UPLOADED FILES

87

UPLOADS AND REVISIONS OF FILES TO THE LMS INSTANCE BY YEAR AND T&L

• The Files area works as a “loading dock” to the particular online course. It enables

the upload of a limited amount of digital files which may be pointed to from the

Pages, Syllabus, Modules, and other sections of the online course.

• Some instructors use the Files area directly for learners to download course information and

datasets.

• In the first four years of the LMS’s use at K-State (we’re in Y5 now), there have been

growing usage of this feature.

• This feature is bolstered by the use of Mediasite as a third-party video hosting site

and player (and desktop lecture capture tool).

89

UPLOADS AND REVISIONS OF FILES TO THE LMS INSTANCE BY YEAR AND T&L (CONT.)


• What sorts of files are being uploaded? Content-wise, is there inherent teaching and

learning value?

• Depending on how the digital contents are harnessed for teaching and learning, what is

the learning value?

90

OBSERVED UPLOADED FILE TYPES AND T&L

• In descending order, the top 10 most popular file types uploaded into this

instance of Canvas were: .docx, .pdf, .jpg, .png, .pptx, .xlsx, .ppt, .zip, .dat,

and .xl. The top 10 file types include text files, image files, slideshow files,

folders of digital contents, data files, and spreadsheet files. Further on, there

are videos, web pages, audio files, and others.

• The uploaded file types may indicate what kinds of technologies learners

have access to in their learning.

92

OBSERVED UPLOADED FILE TYPES AND T&L (CONT.)


• How are the various file types used in respective assignments?

• What is the quality of the online learning contents that learners create?

• Are there ways to increase the multi-modality of the digital objects that learners create?

93

WORD CLOUD OF FILE CONTENTS (FROM THE DESCRIPTIONS OF FILE CONTENTS) AND T&L

• File contents are the files uploaded through the “loading docks” as well as files

uploaded to help create individual profiles within the Canvas LMS (for the particular

instance).

• The word cloud on the prior slide shows the most frequent words mentioned in the

named uploaded files. The assumption is that the file names are informational (and

some are, others not).

• In this case, the long tail may be more informative…since the most frequent words

will be taken up with generic terms labeling what the files may be for.

95

WORD CLOUD OF FILE CONTENTS (FROM THE DESCRIPTIONS OF FILE CONTENTS) AND T&L (CONT.)


• What file naming protocols can instructors use to be as specific and informative about file

contents as possible?

• If the Files area is used by instructors in a published way, what folder structure, folder

naming protocols, and folder names can be used to be as informative as possible?

96

HIGH FREQUENCY WORD COUNTS IN THE FILE NAMES SET (AS ONEGRAMS / UNIGRAMS) AND T&L

Some of the terms, like “reflection” and

“review” are indicative of pedagogical

awareness.

The references to “guide” and others are

also indicative of cognitive scaffolding.

97

8. ABOUT WIKIS AND WIKI PAGES

98

PARENT TYPES FOR WIKI PAGES AND T&L

• Parent types for wiki pages may be either “course” or “group.” A “course”

wiki page points to the pages created at the course level and that may be

shared in a modular or other context. A “group” wiki page points to pages

that learners in groups may have created or co-created. For example,

Groups can have home pages.

100

PARENT TYPES FOR WIKI PAGES AND T&L (CONT.)


• What are the purposes of wiki pages created at the “course” level? What sorts of digital

contents are included in each?

• What are the purposes of wiki pages created at the “group” level? What sorts of

digital contents are included in each?

• What learning activities are related to the respective wiki pages? How effective are

these learning activities in context?

101

WIKI PAGE WORKFLOW AND T&L

• The wiki page workflow state can be in four conditions: active, deleted,

unpublished, and null. “Null” may be a default state when a page exists but

has no contents, and such pages may be pre-made once groups are created,

for example. (The documentation is not clear.)

• An active page is one that has been created and can be viewed.

• A deleted page is one that no longer is available.

• An unpublished page is one that is being drafted.

103

WIKI PAGE WORKFLOW AND T&L (CONT.)


• What are some creative wiki pages that may be an inspiration to other teachers and

learners?

• What are some functions of wiki pages that people do not use often like

• iframes (inline frames)

• embedded video

• third-party software integrations (like Twitter streams), or others?

• How can more creative work on the pages be encouraged?

104

WORD FREQUENCY WORD CLOUD FROM WIKI PAGE TITLES AND T&L

• Wiki page titles refer to the text-based names per each page. Those are required

fields. The word cloud on the prior slide refers to the most frequent terms found in

this text set of titles (treated as a “bag of words”).

• A glance at the word cloud shows action words related to learning: studio, lab, teaching,

project, design, clinical, experience, and public.

• There are subject words, too: physics, chemistry, grain, infant, and others.

• If nothing else, these are suggestive of some of what the pages do.

• In any word frequency count, the “long tail” of few or even single exemplars also will

have value here.

106

WORD FREQUENCY WORD CLOUD FROM WIKI PAGE TITLES AND T&L (CONT.)


• How do these word clouds change over time? Are there differences between these word

frequency counts from wiki page titles year over year? Between departments? Between

domains?

• What about non-English terms in wiki page titles (given the affordances of UTF-8)?

107

9. ABOUT ENROLLMENT ROLE TYPES

108

ABOUT ENROLLMENT ROLE TYPES

Role Name Basic Role Type

Librarian TAEnrollment

StudentEnrollment StudentEnrollment

TeacherEnrollment TeacherEnrollment

TAEnrollment TAEnrollment

DesignerEnrollment DesignerEnrollment

ObserverEnrollment ObserverEnrollment

Grader TAEnrollment

GradeObserver TAEnrollment

109

UNIVERSITY-DEFINED ROLES AND CAPABILITIES AND T&L

• “Enrollment role types” are defined by the university. The “role name” is the

publicly facing side of the role, and the “basic role type” deals with the role-

based functionalities (built on the idea of “least privilege” or “give people

only as much access as they need so as not to compromise security”).

• For example, librarians have as much access as teaching assistants do, based on the

table.

• Do site users have all the access that they need for what they need to do?

• Is the system resilient against potential deletion of data? (sorta)

110

UNIVERSITY-DEFINED ROLES AND CAPABILITIES AND T&L (CONT.)


• Are there super roles that need to be created beyond “admin”?

• Are there more circumscribed roles that need to be created beyond “observer”?

• Are there dedicated roles for one-off applications?

111

FREQUENCIES OF ENROLLMENT ROLES AND T&L

• The treemap on the prior slide shows “frequencies of enrollment roles,” with

the most popular roles in the following descending order: students, teachers,

studentview (observer), teaching assistant, and designer enrollments.

• Are the relative frequency counts / proportions correct?


• Are there new roles that need creating or current roles that need revision or tweaking?

113

TOP DOZEN COMPUTER SYSTEM CONFIGURATIONS FOR ACCESSING LMS INSTANCE AND T&L

• This section captures a large amount of nuance of the computer systems used

to connect to Canvas for the teaching and learning. These inform what

technologies should be used to build digital learning objects, the types of

outputs that should be done, and the accommodations to ensure playability

and accessibility.

• Based on the technologies that people use, digital learning objects need to be

tested on Firefox browser and on mobile devices. However, there’s more…

115

TOP DOZEN COMPUTER SYSTEM CONFIGURATIONS FOR ACCESSING LMS INSTANCE AND T&L (CONT.)


• What are all the main ways that people use to connect to the K-State instance of

Canvas? How can universal design be applied to ensure that they can all access the

contents in the most accessible way possible?

• Also, how can the Canvas apps for iOS and Android be designed to for the optimal

small-screen access?

116

REQUEST TYPES AND T&L

• The most common types of activities on the Canvas site for the K-State

instance is “GET” and “POST.” In other words, people retrieve contents or

access or “read”, and they upload or “create” contents.


• There are two main request types. Are the other types necessary, and if so, how can their

use be encouraged?

118

10. ABOUT GROUPS

119

GROUP NAMES FREQUENCY WORD CLOUD AND T&L

• Instructors use inspiring group names sometimes to rally their learners. The

word cloud on the prior slide shows the most common words found in the text

set of group names.

• Again, it is important to check the “long tail” of resulting pareto chart from

the data table with the word frequency counts.


• Learner motivation is important. What are ways to encourage instructors to have creative

and inclusive / respectful names for learner teams?

121

MODERATOR STATUS OF LEARNERS IN GROUPS AND T&L The “moderator status” refers to assigned

leadership in learner groups. In the K-State

instance, very few instructors have decided to

go with student leaders in the groups,

preferring leadership to emerge (rather than

be assigned), apparently.

123

MODERATOR STATUS OF LEARNERS IN GROUPS AND T&L (CONT.)


• When does it make sense to assign student leaders instead of have them emerge?

• Should “moderators” be trained? Would their work be supervised by the instructor?

• How can leadership be brought into play without learners feeling disenfranchised?

124

LEARNER MEMBERSHIP STATUS IN GROUPS AND T&L

• Based on the numbers in the prior bar chart, learner membership statuses

were either accepted or deleted. There were none in process—none invited

without an answer, and none “requested” without approval or disapproval.

• Instructors may not be using groups that people can apply to membership in,

and if this feature is beneficial to learning, it may be important to explore this

feature and put it into play strategically and tactically.

126

LEARNER MEMBERSHIP STATUS IN GROUPS AND T&L (CONT.)


• How well are groups being used in an LMS? Do they create a sense of camaraderie and

collegiality that my be supportive of learning?

• What is the level of collaborative creative work in learner groups?

• What is the level of discourse in learner groups?

• What are the social dynamics in learner groups?

127

11. ABOUT USERS AND WORKFLOW STATES

128

USER “WORKFLOW” STATES AND T&L

• Users, the people who are in an LMS instance, may have their accounts in one of four

stages: creation_pending, deleted, pre_registered, and registered.

• As may be seen in the prior slide’s piechart, very few are waiting to have their

accounts approved (and there are internal organizational processes for that).

• Some have been deleted (also based on internal policies and practices).

• Some who have been pre-registered, maybe as visiting high school students or

external collaborators on online courses and trainings, are in the instance in not-

insignificant numbers. (It’s not quite clear in the data dictionary what the respective

roles may be.)

130

USER “WORKFLOW” STATES AND T&L (CONT.)


• Are people being processed in- or out- in sufficiently fast and efficient ways for teaching

and learning?

• For learners who need extended access, are there systems and policies and trained

people in place to meet their needs?

131

YEARS OF ORIGINATION OF USER ACCOUNTS AND T&L

• The years of origination show a major push in 2014 to get everyone on the

system.

• Then, it seems that there may be a more baseline of learners going into the

online system.

• Not all students have online accounts, but many F2F courses and blended

courses also use the LMS. Also, there are non-course uses of the LMS.

133

YEARS OF ORIGINATION OF USER ACCOUNTS AND T&L (CONT.)


• Based on knowledge of the campus and its flow of people, how well are they being

integrated into the affordances of the LMS? Is there enough technical support and other

support to ensure that users’ needs are met?

134

RETIRED ACCOUNTS = REGISTERED FALSE AND T&L

• The word cloud on the prior slide gives a light human sense of those who have

been off-ramped from the LMS.

• The uses of the first names make these impossible to re-identify, but data from

the long tail with the rare last names may make these somewhat re-

identifiable (so as always with data, use with care).

136

RETIRED ACCOUNTS = REGISTERED FALSE AND T&L(CONT.)


• Whose accounts are being retired, and why?

137

CREATED PSEUDONYMS AND T&L

• “Pseudonyms” are the “logins associated with users,” which can enable

integrations with other databases to capture information at the individual

human level.

• In the pseudonyms category, there can be other identifiers tied to a person.

• These may be used to anonymize individuals to enable data extraction and

research without the risk of leaking personally identifiable information (PII).

139

CREATED PSEUDONYMS AND T&L (CONT.)


• This area is related more with developer and DBA work.

140

CURRENT “STATES” OF PSEUDONYMS AND T&L

• 97% of pseudonyms are active, and 3% are deleted.

• Proper management of pseudonyms means that those who are active users

should be included, and those who no longer are should have their

pseudonyms deleted, to ensure that everything is clear.

142

CURRENT “STATES” OF PSEUDONYMS AND T&L (CONT.)


• How accurately maintained are the pseudonyms?

• Are they set up as accurately as possible?

143

12. ABOUT COURSE LEVEL GRADES (BASED ON ENROLLMENTS)

144

NUMBERS OF ATTEMPTS FOR LATEST SUBMITTED ASSIGNMENTS AND T&L

• The numbers of attempts on assignments tends towards only one or two

attempts, for assignments that enable more than one submittal.

• A majority are “null,” which may suggest that there is only a one-time

submittal.

146

NUMBERS OF ATTEMPTS FOR LATEST SUBMITTED ASSIGNMENTS AND T&L (CONT.)


• Is it positive or negative to have multiple assignment submittals? In some cases, positive.

In others, negative?

147

13. ABOUT CONVERSATIONS (IN-SYSTEM EMAILS)

148

CONVERSATIONS WITH MEDIA OBJECTS INCLUDED AND T&L

• In Canvas, “conversations” are internal emails within the system.

• It is possible to attach media objects on these emails.

• Only 49/100,000 or 4 in 10,000 conversations in the Canvas LMS instance

contain a media object.

150

CONVERSATIONS WITH MEDIA OBJECTS INCLUDED AND T&L (CONT.)


• When do media objects (audio, video, multimedia, and others) add learning value in

internal emails?

151

CONVERSATIONS W/ OR WITHOUT ATTACHMENTS AND T&L

• 11% of emails in the Canvas instance contain attachments; 89% do not.

• These attachments are any sort of attachment, such as a text file or other file,

including “media objects” (like audio files, video files, etc.).

• Some attachments to “conversations” are assignments to learners from

instructors. Some attachments are homework assignments by learners to

instructors. Some are notes between learners. There are many other use

cases for attachments to conversations in the LMS.

153

CONVERSATIONS W/ OR WITHOUT ATTACHMENTS AND T&L (CONT.)


• Attachments (articles, slideshows, images, and media objects) can often be highly

valuable for learning. How can instructors be encouraged to add these when relevant

(without overwhelming learners with too much information)?

154

ORIGINS OF CONVERSATIONS / MESSAGES AND T&L

• The “origins of conversations” may be either human or system-generated. It is

possible that the auto-generated messaging by email was not turned on,

which explains the reason why there are none.


• Are there potential benefits to auto-generated conversations? (There already are such

emails to @k-state.edu / @ksu.edu emails, but should auto-generated conversations exist

for the LMS instance as well? Or would this just eat up memory and frustrate people?)

156

CONVERSATION MESSAGES WORD FREQUENCY COUNT AND T&L

• What do people in an LMS converse about in the internal email? All sorts of

education concerns, of course (as may be seen in the prior word cloud).


• What communications do learners handle in-LMS vs. through other means (Like university

email systems? Like social media? Like discussion boards? Like web conferencing

sessions?) Are the communications set up as efficiently as possible for learning value?

158

MASS CONVERSATION MESSAGE CONTENTS AND T&L

• In general, at the macro level of the LMS instance over the four years of the analysis,

the conversations tend to be analytical, power-based, and positive in tone, but the

authenticity of the communications tends to be low with authentic meaning “honest,

personal, and disclosing” with higher numbers and “a more guarded, distanced form

of discourse” with lower authentic numbers.


• Is academic speech more analytical, power-based, and positive in sentiment but more

guarded? Maybe. Those who want to succeed may have to be this way.

160

MESSAGING ABOUT “HUMAN DRIVES” IN THE MASS CONVERSATION MESSAGES AND T&L

• The human drives in the expressed conversations on the LMS instance showed

a tendency toward affiliation, power, and reward. There is less push for

achievement and little sense of risk-taking.


• Risk-taking by learners is thought to be positive if it enables people to be more confident

and active in their learning. Should the communications be more achievement oriented

and risk oriented?

162

SENTIMENT ANALYSIS OF SAMPLE OF CONVERSATION MESSAGING AND T&L

• From the sample of conversation messaging, the messages were equal parts

“very positive” and “moderately negative.” Very few messages were “very

negative.”

• It seems constructive that people can be constructively positive and negative.

• The next step would be to analyze the body of the messages that were coded

to the various categories of sentiment.

• Of course, there is a risk of stumbling across sensitive information here in this part of the

LMS data portal.

164

SENTIMENT ANALYSIS OF SAMPLE OF CONVERSATION MESSAGING AND T&L (CONT.)


• There has to be a balance of positive and maybe slightly negative sentiment in the

learning context to help people feel supported and cared about for learning. Instructors

who have been teaching a long time have a sense of the right balance. Looking at the

emails may be too intrusive…but the general point holds that people are more

responsive to positive supports than negative messaging.

165

AUTO-EXTRACTED THEME BASED HIERARCHY CHART OF CONVERSATION MESSAGING SAMPLE & T&L

• Auto-extracted themes (from machine learning) capture a range of topics and

related sub-topics in the email conversations inside the Canvas LMS.

• The treemap shows some of these elements.


• Would online emails change if people knew how observable such conversations were

should anyone be interested?

167

AUTO-EXTRACTED THEMES FROM CONVERSATION MESSAGING SAMPLE AND T&L

• Per the piechart on the prior slide, it looks like the conversations are often

time-based and study-based.


• Some instructors create discussion boards where people may post questions and get

answers back in a timely fashion. Having a central location for questions and answers

may make intercommunications more efficient.

169

CONTEXTS OF “HELP” USE FROM THE CONVERSATION TEXT SET EXPRESSED IN A WORD TREE AND T&L

• It is possible to drill down into a text set to see the context of how a common

term is used (or even uncommon terms). The prior word tree used “help” as a

target term to see the context in which that term was used.


• Are there efficient ways to get help to learners from conversations without the heavy

burden of responding to every student ASAP via email?

171

14. ABOUT THIRD-PARTY EXTERNAL TOOL ACTIVATIONS

172

NUMBERS OF EXTERNAL TOOL ACTIVATIONS AND T&L

• With the advent of Learning Tools Interoperability (LTI), many organizations

and corporations have built tools that interconnect their respective resources

(online software, online services) with LMSes.

• These bridging tools are available to use in the Canvas LMS instance. The

third-party apps are free, but the actual online hosted resources vary in terms

of costs.

• Some resources are open-source, and others are copyrighted. Some are wholly free, and

others require subscriptions or payments.

174

NUMBERS OF EXTERNAL TOOL ACTIVATIONS AND T&L (CONT.)


• What online learning resources would be beneficial to learners? (Khan Academy?

TEDEd? YouTubeED? Google Maps? Google Docs? GitHub?)

175

NAMED EXTERNAL TOOL ACTIVATIONS AND T&L

• The summary treemap shows the various named third-party tool activations in this

instance of Canvas.

• It is possible to use iframe code and embed text to bring in many of the same

resources without having to use the app activation.


• Which tools are being used and how?

• Which third-party apps and tools are being discontinued?

177

EXTERNAL TOOL ACTIVATIONS IN 2013 / 2014 / 2015 / 2016 AND T&L

• What are the external tools being activated based on the alphabetical

histogram bar charts in the prior few slides? What could explain some of the

year-over-year differences?


• How can instructors be encouraged to experiment with third-party apps that can add

functionality and contents to their teaching and learning?

182

15. ABOUT COURSE USER INTERFACE (UI) NAVIGATION ITEM STATES

183

COURSE USER INTERFACE NAVIGATION ITEM STATE AND T&L

• The course user interface navigation item refers to whether an object appears in the

left navigation of the course, whether it is “visible” or “hidden.”

• If this is a summary, it looks like a majority of the available navigational items are

showing; however, this may or may not be positive given that some people do not

hide navigational items that they do not use (and end up confusing students).


• What are ways to encourage instructors to use the LMS to its full functionality but to hide what

they do not use (to support teaching and learning, without adding to the cognitive load)?

185

SUMMARY

186

BACKGROUND KNOWLEDGE REQUIRED

• To activate this LMS data portal data, it helps to know something of the

following:

• The institution of higher education

• What is going on in the front-end of the LMS (through admin access)

• Understandings of online teaching and learning

• Understandings of what happens when a static course goes live with the animating

presences of people and actual teaching

187

CHALLENGES TO USING THIS LMS DATA PORTAL DATA

• So far, it has been

• difficult to make the “business case” for value on campus

• difficult to query for learning value directly (such as learning sequences per learner in a

course)

• difficult political environments on a campus to change instructors’ usages of the LMS

(without stepping on others’ decision-making)

• But this effort has been an informal one…

188

EXTENDING LMS DATA PORTAL DATA FOR TEACHING AND LEARNING

• Improving teaching and learning is hard work. There are political

implications to this effort on a campus (of course).

• Instructors and graduate teaching assistants (GTAs) at the front lines would

enhance the data analyses and are the ones who can most effectively apply

the data to improved teaching and learning; they need to be “onboarded”

for this work.

• The current research insights may be more broadly shared to strengthen the

application of the data.

189

FINDING COMPARABLE EXTERNAL DATA

• While the focus is on institutional improvements and support for teaching and

learning, it is possible to go “macro” and beyond the walls of the

university…to larger contexts.

• To that end, it would benefit to have…

• comparable data from other ~ institutions of higher education (in similar developmental

states of LMS rollout), and

• some sort of comparable baseline data.

190

SOFTWARE USED

• The software used for this presentation include the following: 7Zip, Gadwin

Printscreen, Microsoft Access, SQL Express, LIWC, NVivo 11 Plus, MS Excel

2016, MS Visio, Adobe Photoshop, MS PowerPoint, and others.

191

CONCLUSION AND CONTACT

• Dr. Shalin Hai-Jew

• iTAC, Kansas State University

• 212 Hale Library

• 785-532-5262

• [email protected]

• The data visualizations (by the presenter) come from “Wrangling Big Data in a Small

Tech Ecosystem,” formally published in Summer 2017:

http://scalar.usc.edu/works/c2c-digital-magazine-spring--summer-2017/wrangling-

big-data-in-a-small-tech-ecosystem.

192

mailto:[email protected]

http://scalar.usc.edu/works/c2c-digital-magazine-spring--summer-2017/wrangling-big-data-in-a-small-tech-ecosystem

Using Large-Scale LMS Data Portal Data to Improve Teaching and Learning (at K-State)

Education

Transcript of Using Large-Scale LMS Data Portal Data to Improve Teaching and Learning (at K-State)