FLAN Presentation, 7 June 2016

Evaluating Comments in MOOC Discussion Forums

Tim O’RiordanWeb and Internet Science (WAIS)www.ecs.soton.ac.uk/people/tor1w07 @studywbv

Motivations

Image: Boxing Mail – N.Y. Post Office c. 1910. Library of Congress, NKCR

@studywbv

2

Evaluating Comments in MOOC Discussion

Forums

Content analysis

Image: ALTC2015, Association for Learning Technology ©2015, CC BY-NC-SA 2.0

@studywbv


Forums 3

4

Bloom score

Bloom descriptor

0 - Off-topic There is written content, but not relevant to the subject under discussion.

1 - Remember Recall of specific learned content, including facts, methods, and theories.Verbs: name, describe, relate, find, list, write, tell.

2 - Understand Perception of meaning and being able to make use of knowledge, without understanding full implications. Verbs: explain, compare, discuss, restate, predict, translate, outline.

3 - Apply Tangible application of learned material in new settings.Verbs: show, complete, use, classify, examine, illustrate, implement, solve.

4 - Analyse Deconstruct learned content into its constituent elements in order to clarify concepts and relationships between ideas. Verbs: explain, compare, contrast, examine, identify, investigate, categorise, differentiate, organise.

5 - Evaluate Assess the significance of material and value in specific settings. Verbs: check, decide, rate, choose, recommend, justify, assess, prioritise, critique.

6 - Create Judge the usefulness of different parts of content, and producing a new arrangement.Verbs: synthesise, invent, plan, compose, construct, design, imagine, generate.

A popular and well-respected aid to curriculum development that maps learning to six categories of knowledge acquisition.

Bloom’s Taxonomy


Forums

5

CoI score CoI descriptor

0 - Off-topic There is written content, but not relevant to the subject under discussion.

1 – Triggering event

A contribution that exhibits a sense of puzzlement deriving from an issue, dilemma or problem. Includes contributions that present background information, ask questions or move the discussion in a new direction.Verbs: evoke, induce, contradict

2 - Exploration A comment that is seeking a fuller explanation of relevant information. This can include brainstorming, questioning and exchanging information. Contributions are unstructured and may include: unsubstantiated contradictions of previous contributions, different unsupported ideas or themes, personal stories, and descriptions or facts that are not used as evidence. Verbs: inquire, diverge, search

3 - Integration Previously developed ideas are connected. Contributions include: references to previous messages followed by substantiated agreements or disagreements; developing and justifying established themes; cautious hypotheses; combining different sources; providing a tentative solution to an issue.Verb: test, conjecture, check

4 - Resolution New ideas are applied, tested and defended with real world examples. This involves methodically testing hypotheses, critiquing content in a systematic manner, and expressing supported intuition and insight. Verb: commit, settle, confirm

A highly cited method that categorises 4 phases of discussion.

Cognitive Presence (Community of Inquiry)


Forums

Data

Image: University of Southampton, Understanding Language, Exploring Oceans and Contract Management MOOCs. FutureLearn Ltd 2015

@studywbv

6


Forums

• 500 comments each MOOC• Rated according to 2 methods by 7 raters• Linguistic Inquiry and Word Count (LIWC 2015)• Correlated by:

i) 50 word commentsii) Aggregated batches of 10 contiguous

comments All individual comments

Method@studywbv

7


Forums

Image: University of Southampton, Understanding Language, Exploring Oceans and Contract Management MOOCs. FutureLearn Ltd 2015.

Results

Correlation between Bloom and Cognitive Presence

r = 0.909p = <0.001

@studywbv

8


Forums

Results

Correlation between Bloom and Word Count

r = 0.687p = <0.001

@studywbv

9


Forums

Results

Correlation between Bloom and 1st person singular

r = -0.321p = <0.001

@studywbv

10


Forums

Preliminary ML – CP@studywbv

12


Forums

• 16 attributes:like, word count, WPS, Sixltr, ppron, prep, auxverb, negate, posemo, negemo, cogproc, cause, differ, affiliation, power

• 10 fold Cross-validation

• Classifiers: Naive Bayes, J48, Random Forest

• Accuracy: 60%

• Cohen’s K: 0.48

Preliminary ML@studywbv

13


Forums

Predicted

Actual Off-topic Triggering Exploration

Integration Resolution

Off-topic 135 41 36 6 0

Triggering 41 114 115 18 1

Exploration

13 60 295 92 9

Integration 0 8 120 184 70

Resolution 0 0 9 54 291Confusion matrix for best model – Cognitive presence

@studywbv

[email protected]

Thank you…

www.slideshare.net/studywbv/flan-presentation-7-june-2016

14


Forums

timswww.wordpress.com

FLAN Presentation, 7 June 2016

Education

Transcript of FLAN Presentation, 7 June 2016