Certifying the hard stuff (and where learning analytics might help)

1

Simon Buckingham Shum Knowledge Media Institute, The Open University, UK

simon.buckinghamshum.net learningemergence.net @sbskmi

MOOC Research Conference 2013, Arlington TX Panel on Certifying MOOCs moocresearch.com

content + curriculum

The final educational C… may be about to go Open

community certification

“What you assess is what you get; if you don’t test it, you won’t get it.

To improve student performance we must recognize that essential intellectual abilities are falling through the cracks of conventional testing.”

Lauren Resnick

3

Assessment regimes drive certification

Certification

4

Assumes we know what quality looks like That we know how to assess it, reliably That certification is a brand (but based on scarcity)

The problem of Massive

5

How do we avoid just counting what’s easiest to count?

Big Data and Analytics to the rescue!

6

Finally, we can let the data speak for itself Learning analytics and e-assessment make it possible to objectively evaluate performance at scale

Finally, we can let the data speak for itself Learning analytics and e-assessment make it possible to objectively evaluate performance at scale

Data and Analytics are infused with human values

7

Learning technology is not neutral

8

Any technology embodies epistemological, pedagogical and assessment assumptions “Our learning analytics are our pedagogy” epistemology

pedagogy assessment

Knight S, Buckingham Shum S and Littleton K. (2013) Epistemology, Pedagogy, Assessment and Learning Analytics. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 75-84 Open Access Eprint: http://oro.open.ac.uk/36635

The challenge of authenticity: of learner, and of learning

9

Are you really doing the work? Did you really write this? Plagiarism is not a new problem >>> “authentic assessment”

Authentic Assessment strategies resist plagiarism, promote deep learning and build real world competencies

  Oral Interviews Teacher asks student questions about personal background, activities, readings, and other interests.

  Story or Text Retelling Student retells main ideas or selected details of text experienced through listening or reading.

  Writing Samples Student generates narrative, expository, persuasive, or reference paper with clear reference to their own experience and views.

  Projects/Exhibitions Student works with other students as a team to create a project that often involves multimedia production, oral and written presentations, and a display.

  Experiments/Demonstrations Student documents a series of experiments, illustrates a procedure, performs the necessary steps to complete a task, and documents the results of the actions.

  Constructed-Response Items Student responds in writing to open-ended questions.

  Teacher Observations Teacher observes and documents the students attention and interaction in class, response to instructional materials, and cooperative work with other students.

  Portfolios A focused collection of student work to show progress over time.

10

cf. Jim Groom’s keynote

“In the growth mindset, people believe that their talents and abilities can be developed through passion, education, and persistence …

It’s about a commitment to … taking informed risks … surrounding yourself with people who will challenge you to grow”

Carol Dweck

11 Interview with Carol Dweck: http://interviewscoertvisser.blogspot.co.uk/2007/11/interview-with-carol-dweck_4897.html

Learning dispositions

See the MRI mindset intervention projects

“We’re looking at the profiles of what it means to be effective in the 21st century. […] Resilience will be the defining concept. When challenged and bent, you learn and bounce back stronger.”

“Dispositions are now at least as important as Knowledge and Skills. …They cannot be taught. They can only be cultivated.”

John Seely Brown

12

US Dept. of Educ. http://reimaginingeducation.org conference (May 28, 2013) Dispositions clip: http://www.c-spanvideo.org/clip/4457327 Whole talk: http://www.c-spanvideo.org/program/SecD

Learning dispositions

“It’s more than knowledge and skills.

For the innovation economy, dispositions come into play: readiness to collaborate; attention to multiple perspectives; initiative; persistence; curiosity. The purpose of learning in the 21st century is not to recite inert knowledge but to transform it. It’s time to change the subject.”

13

LearningREimagined project: http://learning-reimagined.com Larry Rosenstock: http://audioboo.fm/boos/1669375-50-seconds-of-larry-rosenstock-ceo-of-hightechhigh-on-how-he-would-re-imagine-learning

Learning dispositions

Larry Rosenstock

How Learning Analytics may help

14

Dispositional Learning Analytics Workshop

15 http://learningemergence.net/events/lasi-dla-wkshp

1st International Workshop on Discourse-Centric Learning Analytics

analytics that look beneath the surface, and quantify linguistic proxies for ‘deeper learning’

Beyond number / size / frequency of posts; ‘hottest thread’

http://ww

w.glennsasscer.com

/wordpress/w

p-content/uploads/2011/10/iceberg.jpg

solaresearch.org/events/lak/lak13/dcla13

Social Network Analysis to gain insight into peer-peer and peer-mentor dynamics (SNAPP tool)

17 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173.

Epistemic Gaming

18 Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Franke, K., Rupp, A. A., Mislevy, R. (2009). Epistemic Network Analysis: A prototype for 21st Century assessment of learning. The International Journal of Learning and Media. 1(2), 33-53.

Badging framework

19 Open Learning Design Studio MOOC — Badging Framework: http://www.olds.ac.uk/blog/moocbadgingandthelearningarc