Participatory Assessment for Engagement, Understanding, and

Achievement in E-Learning Contexts

Daniel Hickey Jenna McWilliams Stephen Bishop Firat Soylu

Learning Sciences ProgramIndiana University School of Education

Thanks!

E-Portfolio Faculty Scholars Program

Proffitt Award Program21st Century Assessment for

Situated and Sociocultural Learning

Indiana Graduate Education Increasingly Dominated by Private Schools

For-Profit Schools are Mostly Online

Institution NameTotal Master's

Degrees Awarded in 2008

Control of institution

Walden University 5007 For-profitUniversity of Phoenix-Online Campus 4027 For-profitGrand Canyon University 3190 For-profitNational University 2344 PrivateLesley University 2035 PrivateNova Southeastern University 1917 PrivateTouro College 1858 PrivateNational-Louis University 1707 PrivateNorthern Arizona University 1369 PublicConcordia University 1249 PrivateMercy College-Main Campus 1140 PrivateTeachers College at Columbia University 994 PrivateMarygrove College 917 PrivateCambridge College 885 PrivateTroy University 867 PublicWilkes University 861 PrivateSaint Mary's University of Minnesota 836 PrivateAzusa Pacific University 818 PrivatePace University-New York 797 PrivateGeorge Mason University 763 Public

Many of the largest providers have significant online or flexible delivery models

available to students. This is clearly linked to the growth in the for-profit market.

Goals for Improving OnlineUniversity Learning

• Balance demands on instructors and designers– Allow more qualified instructors– Reduce costs and increase access– Reduce tension between direct and constructivist methods and

incorporate “connectionist” methods• Increase diverse outcomes

– Group attainment of achievement goals– Individual understanding of targeted concepts– Shared participation in domain discourse

• Enhance validity of outcomes as evidence– Eliminate problem of plagiarism– Preserve validity of achievement outcomes

http://www.iu.edu/~lsmt

Overview of Course Features

• Wikifolios are central course element.– Organized around an individual context– Weekly posting, and commenting

• Groups organized around interests– GroupWiki in final weeks of class

• Reflections on engagement via wikifolios• Exams using multiple choice items– External and textbank items, randomly selected

Social Learning via Wikifolios

Individual Understanding via Reflections

Aggregated Achievement via Tests

Organization of Learning

You can follow along, check back, and discuss by registering at WorkedExamples.org

Research Method• Learning Sciences is interdisciplinary– Defining characteristic is concern with context– Not about design directly (i.e., IST)– Not applied cognitive psychology (i.e., Ed Psych)

• Design-based educational research– Transcends fundamental vs. applied dichotomy– Build “local” theories in refinement contexts– Pay careful attention to contextual framing

• Validation occurs at different levels– Meta-principles (guiding ideas)– Design principles (what ideas mean in contexts)– Specific features (used to enact design principles)

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Learning is primarily communal participation in knowledgeable discourse– Individual learning is secondary (“special cases”)

• Contexts give meaning to concepts– Contextual knowledge is fundamental– This is 21st Century knowledge.

• Activities, artifacts, assessments & accountability are all educational discourses– Represent learning at very different timescales

Participatory Assessment Meta-Principles

Introduction Activate ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Participatory Assessment Design Principles

1. Reframe Concepts:– Treat concepts and skills as tools

2. Scaffold Participation– Wikifolio comments about conceptual tools

3. Assess Reflections– Don’t grade wikifolios directly

4. Control Accountability– Put on-line exams and tests in their place

5. Align Discourse– Refine features to improve participation at all levels

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Contexts give meaning to concepts and skills– “Deep” understanding is relevance across contexts.– Need to develop contextual dispositions

• Range of contexts in online settings– Across projects, across domains, project vs. test, etc.

• Reframe learning as a search for relevance– Each student defines a personally meaningful context– Context used to vigorously frame course reading

#1. Reframe Concepts: Treat Concepts and Skills as Tools

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Learning and Cognition Students Define an Educational Problem or Lesson

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Course Reading as a Search for Relevant Big Ideas (RBIs)

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Broad Coverage via Search for Relevant Specifics

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Foster discourse about relevance in contexts– Wikifolio commenting is more meaningful– Only use discussion forums for logistics

• Pay attention to initial participation– Must enlist tools initially before appropriately– Don’t grade or require posts– Encourage lurking

#2. Scaffold Participation: Wikifolio Commenting on Contextual Relevance

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Personalized Contexts Foster Productive Discourse in Comments

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Personalised Contexts Foster Productive Discourse in Comments

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Personalized Contexts and Lack of Evaluation Can Foster Lots of Discourse

Wikifolio & Comment Characteristics

• Average wikifolio length: 1,580 words– Shortest: 411 words– Longest: 3,899 words

• Number of comments: 1005– Average 62 per week or 3.9 person/week– Average comment length: 120 words– Average 460 additional words per week

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Pattern of Comments in Group

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Distribution of Commenting

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Examples of Consequential Collaboration in Wiki Commenting

• Comments on wikifolios should lead to reflection and revision– Good use of instructor comments.

• Assume that physical and intellectual proximity supports engagement– Overcomes abstract and alienating online context

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Juana’s Example• Chapter 2 Wiki on modal model of memory– Second week of class

• Science Group– Two of the group members are school teachers– One group member was instructional designer

• Juana’s most relevant implication was information processing is easier when to-be-learned information is distributed in working memory

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Linda Comment on Juana’s Wiki

• Linda’s Comment: “The way that I interpreted what you wrote for #1 is that students often have so much information to manage, that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.” (emphasis added)

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Juana’s Later Reflection“After reading everyone comments it seems as though the implications selected are appropriate for my problem except for item number 1 that I have changed from working memory to Resource and Data limitations. (Resource and data limitations constrain learning) This is a much better fit for what my comments were and after rereading that section is clearly the right choice…

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Jeraldine’s Example

• Chapter 2 Wiki• Jeraldine first argued that her most relevant

implication was: All students should be encouraged to ‘manage their resources’

• The comments redirected her thinking about relevance in different contexts.

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Emerald’s Comment

• Emerald’s comment: …almost all of your explanations you wrote about the importance of automaticity, making me wonder why "Automaticity facilitates learning by reducing resource limitations" wasn't your #1 implication. I could understand why you wouldn't choose this if your lesson included more challenging texts, but reading easy texts would require a lot of automaticity even for struggling readers. (emphasis added)

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Jeraldine’s Later Reflection

• Collaborative Reflection: “Emerald commented on my use of automaticity throughout my wiki and wondered why the automaticity implication was not my first choice. I reread my reasoning for my first implication and realized that this is my goal for all my students. However, for my instructional problem, automaticity is the key. Hence, this implication should become my top choice. My second and third implications are almost subheadings of this top implication.”

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

#3. Assess Reflection: Don’t Grade Wikifolios Directly

• Grading artifacts undermines discourse– Need detailed-but-shallow rubrics and exemplars

• Grade reflections as evidence of engagement– Only grade every five weeks– Aim to foster revision– Full points as long as wikifolio was on-time

• Reflections are public too– Fosters learning via sharing while minimizing

plagiarism.

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Assessing Reflection in Wikifolios: Three Types of Engagement

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Assessing Reflection on WikifoliosTypical Reflection

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

#4. Control Accountability:Put Tests in their Place

• Downplay importance of tests– Align with discussion and reflections– Engagement should ensure passing score

• Use time-limited multiple-choice tests– Focus accountability on measurable procedural &

conceptual engagement• Include consequential & critical engagement?– Non-prompted engagement reveals dispositions– May be more appropriate for quizzes

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Use design-based research methods– Refine features to validate specific principles.– Build local rather than fundamental theory

• Align participation to raise all outcomes– Preserve evidential validity by changing format

• Three or more increasingly formal cycles– Design (design activities and artifacts)– Implementation (refine activities & artifacts)– Evaluation (document formal learning outcomes)

#5. Align DiscourseRefine Participation at all Levels

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection  

Social Learning via Wikifolios

Individual Understanding via Reflections

Aggregated Achievement via Tests

Available Spring 2011 from Indiana UniversityOnline Courses in Learning Sciences, Media, and Technology (LSMT)

All Courses Taught by IU College of Education Faculty

P574: Computational Technologies in Educational Ecosystems

Survey of how different technologies ranging from games to simulations to blogging can support learning in variety of contexts, ranging from science classrooms to informal after-school programs and homes.

Dr. Joshua Danish, Assistant Professor, Finalist, Sakai

Teaching with Technology Award, 2010

P507: Assessment in Schools

Learn to use a range of formative and summative assessments in a variety of contexts. Each student will focus deeply on a particular context and domain; the class will learn broadly by contrasting different contexts.

Dr. Daniel Hickey, Associate Professor, International Leader in Assessment

P574: Learning in New Media

A hands-on studio art approach explores how current literacy theory and research in arts education, media education, and computer science applies to learning in new media.

Dr. Kylie Peppler, Assistant Professor,2009 Indiana Governors Award for Tomorrow’s Leaders.

Thank You!

• Feel free to contact me– dthickey@indiana.edu or 812-322-3436

• LMST Program – www.iu.edu/~lsmt

• Learning Sciences Program – www.iu.edu/~learnsci

• Worked Example Fixing E-Learning– www.workedexamples.org