Monitoring Conceptual Development
-
Upload
fridolinwild -
Category
Education
-
view
525 -
download
0
description
Transcript of Monitoring Conceptual Development
![Page 1: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/1.jpg)
Monitoring Conceptual Development
Fridolin Wild, Debra Haley, Katja BuelowKMi, The Open University
ICWL, Shanghai, China, December 9th, 2010
![Page 2: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/2.jpg)
Concepts things we can (easily) learn from
or express in language
• Tying shoelaces• Douglas Adams’
‘meaning of liff’:– Epping: The futile movements
of forefingers and eyebrows used when failing to attract the attention of waiters and barmen.
– Shoeburyness: The vague uncomfortable feeling you get when sitting on a seat which is still warm from somebody else's bottom
I have been convincingly
Sapir-Whorfed by this book.
![Page 3: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/3.jpg)
Semantic = Meaning = …
![Page 4: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/4.jpg)
Meaningful Interaction Analysis
Two-Mode factor analysis of the co-occurrences in the
terminologyResults in a latent-semantic
vector spaceWhich can be analysed with
Network Analysis
![Page 5: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/5.jpg)
The mathemagics behindMeaningful Interaction Analysis
disambiguation with context heterogeneous corpus
![Page 6: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/6.jpg)
The mathemagics behindMeaningful Interaction Analysis
associative closenessmeaning space
![Page 7: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/7.jpg)
The mathemagics behindMeaningful Interaction Analysis
network analysis is used to identify communities of related understanding
![Page 8: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/8.jpg)
Usage Example: Reflecting on Conceptual Change
Reflection is an interactive process of creative sense-making
of the past
![Page 9: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/9.jpg)
Capturing traces in text
![Page 10: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/10.jpg)
Internal latent-semantic graph structure (MIA output)
![Page 11: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/11.jpg)
Software Support: Conceptual Inspection Analytics
![Page 12: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/12.jpg)
Emergent Reference Models
![Page 13: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/13.jpg)
Evaluation
• Evaluating effectiveness: measure of the accuracy in representing conceptual development
• Can be measured with two complementary methods by assessing the external validity of:– Concept Annotation: effectiveness in selecting
accurate conceptual descriptors (with ratings)– Concept Proximity: effectiveness in representing
proximity (with card-sorts)– By comparing against human ratings
of 18 first-year medical students of the University of Manchester Medical School aged 19-21
![Page 14: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/14.jpg)
Concept Annotation• Annotation of 5 authentic
postings again on ‘safe prescribing’
• Selection of 10 top-loading concepts
• Adding of 5 random distracters• Participants ranked on Likert
Scale of 1 to 5how good the concept described the posting
• Human Interrater Correlation was measured with free marginal Kappa (Randolph, 2005)
Conflated categories (1+2,3,4+5)
![Page 15: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/15.jpg)
Concept Proximity (1)• Four authentic learner blog
postings about ‘safe prescribing’ generated ~ 50 top-loading concepts each
• Printed on cards• Participants grouped
them in piles• Comparison of participant
clustering with kmeans-based clustering in the MIA space
• 1% of term pairs put into same cluster by more than 12 participants
• 7% by between 7 and 12
1% term pairs: Spearman’s Rho as interrater correlation
![Page 16: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/16.jpg)
Proximity (2)
![Page 17: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/17.jpg)
Proximity (3)
• Silhouette width in the MIA space
Silhouette plots depict for each observation, how good the balance between its distances to its other cluster members compared to its distances within the next close cluster is.
(Rousseeuw, 1986)
![Page 18: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/18.jpg)
Conclusion• 1st year students do not have much agreement in rating the
annotations, could be a sign of heterogeneous frames of reference
• Activation strength of the 10 concepts has not been taken into account (would be interesting!)
• Still: pretty good clustering results in the upper range• Lower range: could be an artefact of the clustering (clustering of
a folded-in posting, not clustering in the space)• All in all: points towards rigorous use of thresholds• Near human results (at the human overlap)• Near human results (producing clearly better results than chance,
but no perfect agreement)
![Page 19: Monitoring Conceptual Development](https://reader033.fdocuments.in/reader033/viewer/2022061112/545753b2af795900058b6e7e/html5/thumbnails/19.jpg)
The End