ASSESSMENT CULTURE AND INFORMATIONAL SELF-DETERMINATION: A COMPARATIVE STUDY OF ELECTRONIC PERFORMANCE MONITORING IN ONLINE LEARNING ENVIRONMENTS IN THE U.S. AND GERMANY

Daniel Knox: The University at Albany, State University of New York, U.S.

Markus Deimann: Der FernUniversität in Hagen, Germany

Roadmap

Conceptual frame Culture of Assessment Informational Self-Determination

Data & Methods Preliminary Results Implications for Research & Practice

Issue and Purpose

Culture of Assessment

Proponents Institutions should commit to assess student learning

outcomes (Michaud, 2010). (Typically) quantitative methods over qualitative

methods (Turgeon, 2010). Formative assessments over summative assessments

(Weiner, 2009).

Culture of Assessment

Critics

Assessment as EPM

Online assessment is electronic performance monitoring (EPM): The use of computer and communication technologies to collect and store

information about individual or group performance (Aiello & Douthitt, 2001)

Two distinct aspects: Automated Totalizing

“Blackboard records every click by a user within a course and allows instructors to generate graphical reports on course usage and actvity” (UWE Staff Guide, 2009).

Informational Self-Determination

Legal concept: The degree to which a subjects maintain control of their

own data (Hornung, 2009). Distinct from the U.S. concept of privacy.

Legal Origins: Volkszählungsurteil (census verdict), December 15,

1983 Personal data in Germany are constitutionally protected. Individuals have the power to decide when and to what

extent personal information is viewed.

Informational Self-Determination

Implications for online learning: Data Protection Commissioner at every institution. Place strict limitations in the collection and storage of

personal information. Every automated act of data processing that can be directly

linked to a certain person must be regulated by a specific law. Otherwise data processing must be authorized in each instance

by the individual.

Informational Self-Determination

In practice: In online, asynchronous courses, German faculty may

not: Require participation in discussions. Grade discussions posts.

Creates uncertainty on the part of instructors.

The Culture of Assessment vs. Informational Self-Determination

Not universal ‘best practices,’ but culturally embedded.

Both encroach upon academic freedom. So…what do faculty do to promote engagement in

online discussions?

SLN Data & Methods

Data: Stratified random sample of 160 asynchronous, fully-online

courses drawn from the State University of New York Learning Network’s (SLN) archives.

Methods: Course documents (syllabi) analyzed for references for

online discussion policies (1141 of 2058 documents) and compiled.

Discussion policy documents analyzed using grounded theory methodology (Strauss & Corbin, 1998).

Each sentence analyzed using the message as the unit of analysis in Atlas.ti.

The Culture of Assessment Goes Online

Absence of a critical stance. Universality of practices.

National Research Council’s Commission on Behavioral and Social Sciences and Education How People Learn (Bransford et. al. 2000)

“Good learning environments are knowledge centered, learner centered, and assessment centered.”

“Effective teachers … do a great deal of on-line monitoring of both group work and individual performances…” (pg. 140).

Community of Inquiry (Anderson, Rourke, Garrison, & Archer, 2001 )

FernUniversität Data & Methods

Internet survey 20 item questionnaire 154 faculty from cultural and social sciences Delivered via LimeSurvey 25 responses (16.2%)

Results SLNQuantitative Properties (9) Quantitative Dimensions (5)General participation Hard countInitial posting MinimumLead discussion MaximumLines OptimumLogins RangeLogins with postRespond to initial postsRespond to peersWordsQualitative Properties (12) Qualitative Dimensions (4)Accuracy UndefinedCreativity GeneralCritical Thinking SpecifiedEtiquette Highly SpecifiedGeneral InteractivityIntegrityObjectivityReference PeersRelevanceSubjectivitySupporting EvidenceWriting

SLN Example

Monitored properties and dimensions in a graduate education course:

Property Presence Dimension Number Low Range High Range UnitQuantitative YesGeneral participation Property Not FoundInitial posting Property Not FoundLead discussion Property Not FoundLines Property Not FoundLogins Property Found Minimum 2 WeekLogins with post Property Not FoundRespond to initial posts Property Found Minimum 2 ModuleRespond to peers Property Found Hard Count 1 DiscussionWords Property Not Found Presence Specification Qualitative YesAccuracy Property Not FoundCreativity Property Not FoundCritical Thinking Property Found GeneralEtiquette Property Found SpecifiedGeneral Interactivity Property Not FoundIntegrity Property Not FoundObjectivity Property Not FoundReference Peers Property Not FoundRelevance Property Not FoundSubjectivity Property Found GeneralSupporting Evidence Property Found SpecifiedWriting Property Not Found

FernUniversität Results

Content Introduce provocative topics/material Pose challenging questions Refer to previous questions Assignments with a practical orientation for the students

Structure Small group discussion for low participating students. Use the LMS environment to present discussion as a ‘normal’ part of online learning.

Modeling Posting by instructor to begin the course. Explain interactions and advantages of participation. Explain relevance. Use peer tutoring and mentoring to model online discourse.

Conclusion

Implications for research ‘Universal’ assessment best practices should be tested for validity (and

appropriateness) across different cultures. Develop and test alternate models of assessment.

Implications for practice Develop culturally sensitive monitoring policies and incorporate into

faculty training. Instructors should consider opening a dialogue with students about the

topic.

References

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2) Retrieved December 10, 2004, from http://www.aln.org/publications/jaln/v5n2/

Aiello, J. R., & Douthitt, E. A. (2001). Social Facilitation from Triplett to Electronic Performance Monitoring. Group Dynamics: Theory, Research, and Practice, 5(3), 163-180.

Bransford, J.D., A. L. Brown, and R.R. Cocking, eds. (2000). How people learn: brain, mind, experience, and school. Washington, D.C.: National Academy Press, 206-230.

Michaud, Olivier (2010). The effects of assessment: A reflection from within the economic worldview in education. Analytic Teaching and Philosophical Praxis. 30 (1): 20-30.

Strauss, A., & Corbin, J. (1998). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc.

Turgeon, Wendy C. (2010). Confessions of a department chair of assessment. Analytic Teaching and Philosophical Praxis. 30 (1): 12-19.

University of Western England (2009). Blackboard Staff Guide: Tracking and Statistics. University of Western England, Bristol

Weiner, Wendy F. (2009). Establishing a culture of assessment: Fifteen elements of assessment success – how many does your campus have? Academe Online (July-August). http://www.aaup.org/AAUP/pubsres/academe/2009/JA/Feat/wein.htm