Epistemology, Pedagogy, Assessment – Learning Analytics?

Simon Knight @sjgknightImage from http://xkcd.com/903/ licensed under a Creative Commons Attribution-NonCommercial 2.5 License.

Acknowledgements

Thanks to my co-authors and supervisors Simon Buckingham Shum and Karen Littleton for their work on this and our workshop papers

Thanks to Cindy Kerawalla and anonymous reviewers for their helpful suggestions

Images – mostly from Wellcome Images http://wellcomeimages.org/ under CC licence

LÆAP-ing the divide1. LA ‘buy in’ to ways of

thinking about epistemology, assessment and pedagogy

2. For theoretical, practical, ethical reasons we should engage in these debates

3. These considerations have practical implications – the middle ground

Talk Structure

1. Introduce epistemology2. Discuss implications for

assessment (& LA) – the Danish example

3. The other side of the coin – application & understanding student’s processes

4. An epistemology for knowledge building (tools)

Introduction to Epistemology, Pedagogy & Assessment

The TriadForeground relationships between:• epistemology (the nature of knowledge)• assessment (of learnt knowledge?)• pedagogy (the nature of learning)

Key questions• What does it mean to

know?• How do we decide

(assess) if someone knows or not?

• How do we get people to come to know (to learn)?

Why does epistemology matter?“…assessment is one area where notions of truth, accuracy and fairness have a very practical purchase in everyday life”

(Williams, 1998, p. 221).

Why does epistemology matter?“…assessment is one area where notions of truth, accuracy and fairness have a very practical purchase in everyday life”

(Williams, 1998, p. 221).

• LA ‘buy in’ to particular ways of thinking about these issues, but they might be flexible enough to move beyond the current impasse

Learning Analytics

• Data mining• Digital trace –

including linguistic data

• Deployed for pedagogic purposes

Epistemology & Assessmentan example

Danish exams with internet access

• Allows testing of problem-solving and analysis - sifting information

• "if you allow communication, discussions, searches and so on, you eliminate cheating because it's not cheating any more. That is the way we should think."

• Potential for auto-grading – LA role, a new model?http://www.timeshighereducation.co.uk/416090.article

http://news.bbc.co.uk/1/hi/education/8341589.stm

Epistemological assumptions

A role for LA?

This example illustrates…Epistemology of assessment, through exploring student’s judgements on epistemic concepts such as ‘credibility’

Epistemological assumptions

- Knowledge is communicative, and discursive; language both represents, and creates knowledge

- Separating nuggets of information from their contextualised uses may not assess knowledge

Assessment should nurture knowledge development and practices (through AfL)

Pedagogic assumptions

• Assessment is used in teaching (but doesn’t drive it)• Pedagogy should involve knowledge practices – not

assessment practices• Discourse is fundamental

The other side of the coin

The other side of the coin…

• What we ask students to do• What they do

Epistemology: In action• “What they do”• Analytics give unprecedented(?) access• Marks a shift, from standardised assessments

to knowledge in action• From seeing in a vacuum, to seeing “for”

“epistemological beliefs are a lens for a learner's views on what is to be learnt” (Bromme, 2009)

• Certainty, simplicity, source, justification – for knowing• Search allows us to probe; Collaborative Information Seeking

is particularly salient

The Lens of Epistemic Beliefs Behaviours

No ‘unveiling’ of beliefs

• “In action” is key; language “to do”• Psychometrics - many issues of standardised

assessments (& current measures are poor)• They mask variance, subtleties and contexts

Examples of epistemic behaviour

Explicitly or implicitly (behaviour not belief…) people:• select sources of information• have standards by which to judge credibility• collate information in meaningful ways• build arguments and make claims• decide when to start and stop looking for

information• decide (implicitly or explicitly) the breadth and

depth of information required

LA in Structured Knowledge Building

• Strong CSCL tradition to ‘make explicit’ in structured environments (Knowledge Forum, Belvedere, etc.)

Evidence Hub

Epistemic Behaviours & ToolsBy sharing such maps we:– Facilitate in-team information management by

making sensemaking explicit– Can capture information about domain structures

for future users in distributed, asynchronous or implicit information seeking

– Can feedback to users to support more advanced epistemic perspectives (e.g. seeking evidence)

Next ‘class’ - LÆAP the divide • LA built on well grounded EAP• Importance of dialogue around knowledge

building• The methods to analyse this meaningfully (see

also DCLA paper)

Dialogue as a frame• Moving beyond ‘what do they access/click/add’• Dialogue creates co-constructed frames for knowledge

building, and problem identification• Pragmatic, sociocultural context

Natural Language Processing based LA

• Discursive context• Discourse both as a context, and as a creator

of context – historic and fluid properties• Importance of dialogue in knowledge creation• Systems provide a target for chat, and a means

to mark sections of chat

Discourse Analysis

• Exploratory and accountable talk as social mode of thinking or interthinking

• …with epistemic implications• Foreground epistemic language used to

indicate the perspectives taken on content

Analytics questions addressed:

• How do users interthink to construct and solve problems?

• How do users co-conceptualise their information needs, and how good are they at finding and mapping information to those needs?

• How do users use each other, and information, to build knowledge together

LÆAP into the middle space1. LA implicate epistemologies

(Danish e.g.)2. Use of data for particular

conversations/assessment/ AfL, is key

3. Design implications – foreground particular facets of data & activity

Thank you

@sjgknightsjgknight@gmail.comhttp://people.kmi.open.ac.uk/knight/

Tools:• http://www.cs.waikato.ac.nz/ml/weka/index.html• http://mallet.cs.umass.edu/• http://solar.evidence-hub.net/ • http://cohere.open.ac.uk/

Our papers in this area:• Discourse, computation and context – sociocultural DCLA revisited

http://oro.open.ac.uk/36640/ • Tracking epistemic beliefs and sensemaking in collaborative

information retrieval http://oro.open.ac.uk/36553/