Participatory Assessment for Engagement, Understanding, and Achievement in E-Learning Contexts...
-
Upload
reginald-james -
Category
Documents
-
view
213 -
download
0
Transcript of Participatory Assessment for Engagement, Understanding, and Achievement in E-Learning Contexts...
Participatory Assessment for Engagement, Understanding, and
Achievement in E-Learning Contexts
Daniel Hickey Jenna McWilliams Stephen Bishop Firat Soylu
Learning Sciences ProgramIndiana University School of Education
Thanks!
E-Portfolio Faculty Scholars Program
Proffitt Award Program21st Century Assessment for
Situated and Sociocultural Learning
Indiana Graduate Education Increasingly Dominated by Private Schools
For-Profit Schools are Mostly Online
Institution NameTotal Master's
Degrees Awarded in 2008
Control of institution
Walden University 5007 For-profitUniversity of Phoenix-Online Campus 4027 For-profitGrand Canyon University 3190 For-profitNational University 2344 PrivateLesley University 2035 PrivateNova Southeastern University 1917 PrivateTouro College 1858 PrivateNational-Louis University 1707 PrivateNorthern Arizona University 1369 PublicConcordia University 1249 PrivateMercy College-Main Campus 1140 PrivateTeachers College at Columbia University 994 PrivateMarygrove College 917 PrivateCambridge College 885 PrivateTroy University 867 PublicWilkes University 861 PrivateSaint Mary's University of Minnesota 836 PrivateAzusa Pacific University 818 PrivatePace University-New York 797 PrivateGeorge Mason University 763 Public
Many of the largest providers have significant online or flexible delivery models
available to students. This is clearly linked to the growth in the for-profit market.
Goals for Improving OnlineUniversity Learning
• Balance demands on instructors and designers– Allow more qualified instructors– Reduce costs and increase access– Reduce tension between direct and constructivist methods and
incorporate “connectionist” methods• Increase diverse outcomes
– Group attainment of achievement goals– Individual understanding of targeted concepts– Shared participation in domain discourse
• Enhance validity of outcomes as evidence– Eliminate problem of plagiarism– Preserve validity of achievement outcomes
Overview of Course Features
• Wikifolios are central course element.– Organized around an individual context– Weekly posting, and commenting
• Groups organized around interests– GroupWiki in final weeks of class
• Reflections on engagement via wikifolios• Exams using multiple choice items– External and textbank items, randomly selected
Social Learning via Wikifolios
Individual Understanding via Reflections
Aggregated Achievement via Tests
Organization of Learning
You can follow along, check back, and discuss by registering at WorkedExamples.org
Research Method• Learning Sciences is interdisciplinary– Defining characteristic is concern with context– Not about design directly (i.e., IST)– Not applied cognitive psychology (i.e., Ed Psych)
• Design-based educational research– Transcends fundamental vs. applied dichotomy– Build “local” theories in refinement contexts– Pay careful attention to contextual framing
• Validation occurs at different levels– Meta-principles (guiding ideas)– Design principles (what ideas mean in contexts)– Specific features (used to enact design principles)
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
• Learning is primarily communal participation in knowledgeable discourse– Individual learning is secondary (“special cases”)
• Contexts give meaning to concepts– Contextual knowledge is fundamental– This is 21st Century knowledge.
• Activities, artifacts, assessments & accountability are all educational discourses– Represent learning at very different timescales
Participatory Assessment Meta-Principles
Introduction Activate ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Participatory Assessment Design Principles
1. Reframe Concepts:– Treat concepts and skills as tools
2. Scaffold Participation– Wikifolio comments about conceptual tools
3. Assess Reflections– Don’t grade wikifolios directly
4. Control Accountability– Put on-line exams and tests in their place
5. Align Discourse– Refine features to improve participation at all levels
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
• Contexts give meaning to concepts and skills– “Deep” understanding is relevance across contexts.– Need to develop contextual dispositions
• Range of contexts in online settings– Across projects, across domains, project vs. test, etc.
• Reframe learning as a search for relevance– Each student defines a personally meaningful context– Context used to vigorously frame course reading
#1. Reframe Concepts: Treat Concepts and Skills as Tools
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Learning and Cognition Students Define an Educational Problem or Lesson
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Course Reading as a Search for Relevant Big Ideas (RBIs)
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Broad Coverage via Search for Relevant Specifics
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
• Foster discourse about relevance in contexts– Wikifolio commenting is more meaningful– Only use discussion forums for logistics
• Pay attention to initial participation– Must enlist tools initially before appropriately– Don’t grade or require posts– Encourage lurking
#2. Scaffold Participation: Wikifolio Commenting on Contextual Relevance
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Personalized Contexts Foster Productive Discourse in Comments
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Personalised Contexts Foster Productive Discourse in Comments
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Personalized Contexts and Lack of Evaluation Can Foster Lots of Discourse
Wikifolio & Comment Characteristics
• Average wikifolio length: 1,580 words– Shortest: 411 words– Longest: 3,899 words
• Number of comments: 1005– Average 62 per week or 3.9 person/week– Average comment length: 120 words– Average 460 additional words per week
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Pattern of Comments in Group
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Distribution of Commenting
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Examples of Consequential Collaboration in Wiki Commenting
• Comments on wikifolios should lead to reflection and revision– Good use of instructor comments.
• Assume that physical and intellectual proximity supports engagement– Overcomes abstract and alienating online context
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Juana’s Example• Chapter 2 Wiki on modal model of memory– Second week of class
• Science Group– Two of the group members are school teachers– One group member was instructional designer
• Juana’s most relevant implication was information processing is easier when to-be-learned information is distributed in working memory
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Linda Comment on Juana’s Wiki
• Linda’s Comment: “The way that I interpreted what you wrote for #1 is that students often have so much information to manage, that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.” (emphasis added)
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Juana’s Later Reflection“After reading everyone comments it seems as though the implications selected are appropriate for my problem except for item number 1 that I have changed from working memory to Resource and Data limitations. (Resource and data limitations constrain learning) This is a much better fit for what my comments were and after rereading that section is clearly the right choice…
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Jeraldine’s Example
• Chapter 2 Wiki• Jeraldine first argued that her most relevant
implication was: All students should be encouraged to ‘manage their resources’
• The comments redirected her thinking about relevance in different contexts.
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Emerald’s Comment
• Emerald’s comment: …almost all of your explanations you wrote about the importance of automaticity, making me wonder why "Automaticity facilitates learning by reducing resource limitations" wasn't your #1 implication. I could understand why you wouldn't choose this if your lesson included more challenging texts, but reading easy texts would require a lot of automaticity even for struggling readers. (emphasis added)
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Jeraldine’s Later Reflection
• Collaborative Reflection: “Emerald commented on my use of automaticity throughout my wiki and wondered why the automaticity implication was not my first choice. I reread my reasoning for my first implication and realized that this is my goal for all my students. However, for my instructional problem, automaticity is the key. Hence, this implication should become my top choice. My second and third implications are almost subheadings of this top implication.”
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
#3. Assess Reflection: Don’t Grade Wikifolios Directly
• Grading artifacts undermines discourse– Need detailed-but-shallow rubrics and exemplars
• Grade reflections as evidence of engagement– Only grade every five weeks– Aim to foster revision– Full points as long as wikifolio was on-time
• Reflections are public too– Fosters learning via sharing while minimizing
plagiarism.
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Assessing Reflection in Wikifolios: Three Types of Engagement
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Assessing Reflection on WikifoliosTypical Reflection
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
#4. Control Accountability:Put Tests in their Place
• Downplay importance of tests– Align with discussion and reflections– Engagement should ensure passing score
• Use time-limited multiple-choice tests– Focus accountability on measurable procedural &
conceptual engagement• Include consequential & critical engagement?– Non-prompted engagement reveals dispositions– May be more appropriate for quizzes
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
• Use design-based research methods– Refine features to validate specific principles.– Build local rather than fundamental theory
• Align participation to raise all outcomes– Preserve evidential validity by changing format
• Three or more increasingly formal cycles– Design (design activities and artifacts)– Implementation (refine activities & artifacts)– Evaluation (document formal learning outcomes)
#5. Align DiscourseRefine Participation at all Levels
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Introduction Reframe ConceptsScaffold
ParticipationAlign Discourse
Control Accountability
Assess Reflection
Social Learning via Wikifolios
Individual Understanding via Reflections
Aggregated Achievement via Tests
Available Spring 2011 from Indiana UniversityOnline Courses in Learning Sciences, Media, and Technology (LSMT)
All Courses Taught by IU College of Education Faculty
P574: Computational Technologies in Educational Ecosystems
Survey of how different technologies ranging from games to simulations to blogging can support learning in variety of contexts, ranging from science classrooms to informal after-school programs and homes.
Dr. Joshua Danish, Assistant Professor, Finalist, Sakai
Teaching with Technology Award, 2010
P507: Assessment in Schools
Learn to use a range of formative and summative assessments in a variety of contexts. Each student will focus deeply on a particular context and domain; the class will learn broadly by contrasting different contexts.
Dr. Daniel Hickey, Associate Professor, International Leader in Assessment
P574: Learning in New Media
A hands-on studio art approach explores how current literacy theory and research in arts education, media education, and computer science applies to learning in new media.
Dr. Kylie Peppler, Assistant Professor,2009 Indiana Governors Award for Tomorrow’s Leaders.
Thank You!
• Feel free to contact me– [email protected] or 812-322-3436
• LMST Program – www.iu.edu/~lsmt
• Learning Sciences Program – www.iu.edu/~learnsci
• Worked Example Fixing E-Learning– www.workedexamples.org