Developing Digital Literacy:Baselining and evaluationHelen Beetham, Synthesis Consultant

Jay Dempster, Evaluation Consultant

4th October 2011

Maple House, Birmingham

Welcome – who we areDigital Literacy

Helen BeethamSynthesis consultant

Jay DempsterEvaluation consultant

Session outlineDigital Literacy

Why baseline?The programme-level view

The project-level view (1):How will practice/provision/literacy be developed?

The project-level view (2):Collecting and making sense of baseline data

Programme tools and resources for baselining

Why baseline?Digital Literacy

At programme level:Provide snapshot of the sector at start-upDevelop shared understandingReview what we already know and have achievedContextualise programme aims and rationale

At project level:Provide starting-point for evaluationProvide clear structure for collecting evidenceRefine objectives and success indicators

What do we know?

2008 Learners' experiences of e-learning programmeStudents' success depends on strategies for integrating ICT into academic practice; strategies and preferences differ widely

2009 Learning Literacies for a Digital Age studyDigital literacy needs to be integrated across the curriculum: learners develop through authentic tasks in meaningful situations

2010 Supporting Learners in a Digital AgeInstitutions are: addressing skills and expectations at transition; creating digitally-enabled learning spaces; listening to learners; embedding digital activities into the curriculum

2011 Digital literacy workshop seriesLessons and recommendations, tools for organisational and curriculum development; sharing best practice

2011-13 Developing Digital Literacies programme12 institutions, taking different views across the DL landscape

Digital Literacy

Why digital literacy?Digital Literacy

Impacts of digital technologyon practice

(research, teaching, learning, professionalism, knowledge transfer,

development)

New demands on the sector (graduate attributes and outcomes,

the learning experience, lifelong learning)

Fair access and opportunity(digital technology reduces some

barriers/inequalities, can introduce others)

Addressing digital literacy...Digital Literacy

inhigher/further education (curriculum development,

learning skills/academic practice)

through higher/further education (employability, graduate attributes,

the co-curriculum)

into higher/further education(transition, entitlement, access, student markets, participation)

Whose digital literacy?Digital Literacy

How do we conceive of digital literacy?Digital Literacy

It will be a big challenge to identify how digital literacy makes a difference to the individual in terms of how they think and what they do.Joe Nicholls, Digidol

We need to know more about the messy, day-to-day lived practices of students and staff.Lesley Gourlay, Institute of Education

How do we conceive of digital literacy?Digital Literacy

Presentation, Production and Performanceplay, performance, simulation, appropriation

Collaboration and work skillsmultitasking, distributed cognition, collective intelligence

Participating in information practices transmedia navigation, networking, negotiation

Ryberg and Georgsen (2010), drawing on Buckingham (2005) and Jenkins (2006)

At what level?Digital Literacy

access

skills

practices

identities 'I am...'

'I do...'

'I can...'

'I have...'

(Beetham and Sharpe 2009)

Discipline/subject

specialist focus

Generic/entitlement

focus

With what institutional impact?Digital Literacy

With what focus?Digital Literacy

(Sharpe et al. 2011)

Working with which professionals?Digital Literacy

Digital Literacy

'If we all do it properly, and climb and stay out of our bunkers, this will be a cross-community project on a

grand and productive scale'David Baume, SEDA

Activity in projectsDigital Literacy

Review some of the different ways of 'developing digital literacies'

Which have resonance for your project?

What part(s) of the landscape will your project focus on?

Write down three ways in which your project will bring about development

what will be developed (people, places, practices, processes, policies, curricula, opportunities...) and how?

Activity in projectsDigital Literacy

Review some of the different ways of 'developing digital literacies'

Which have resonance for your project?

What part of the landscape is your project focusing on?

Write down three ways in which your project will bring about development, i.e. what will be developed (people, places, practices, processes, policies, curricula, opportunities...) and how.

Share your statements with another project, working in project pairs. Review, compare, critique.

Baselining

From the original circular:

“Review current processes and practices to establish a baseline description of current digital literacy provision to aid in the evaluation of subsequent innovations.”

This aims to provide the evidence and ideas to support your project in planning your interventions.

Baseline report by end Dec/Jan

Many projects are looking to do a more detailed review by end of year 1.

Baselining as...

stakeholder review for shared/distinct meanings

contextualising ‘big picture’ evaluation questions

provide clear structure for collecting evidence

checking back against the ‘why and how’ of project

achievements

‘Baselining for success’

Combine your 30,000 ft view activity with 'what success looks like‘ ...

produce a ‘rich picture’ of your project’s intended outcomes

identify some clear indicators of success for your different stakeholder audiences

clarify starting points for sources & types of data for your evidence base

hard metrics e.g. statistics in audit documents, quantitative data collected from student feedback surveys

vs. narrative descriptions e.g. example in CDD of a project (T-Sparc) doing an entire baseline as a series of video clips .... on the Design Studio website at: http://blogs.test.bcu.ac.uk/tsparc/t-sparc-baseline-review/

Baselining data and indicators of success

Baselining Digital Literacies

Share your initial ideas/issues in pairs & discuss ...

for instance, you might touch on:

1. Is there a shared sense of what digital literacies mean/are?

2. Based on what/whose evidence?

3. How are you triangulating/cross-checking with stakeholders?

4. What are the assumptions made about the project and its context?

5. Are there existing measures available from prior audits/surveys/initiatives?

6. What descriptors/metrics/rubrics are emerging?

(towards a common focus and common language for how we define and measure

effectiveness & progress in digital literacies)

1. What are the most useful/feasible baseline data/ indicators of success for YOUR project?

Tools & Resources

Tools & Resources

http://bit.ly/pHxQnS

Design Studio pages for JISC DLL:http://jiscdesignstudio.pbworks.com/w/page/46421608/Developing%20digital%20literacies

Support Wiki pages for Baselining:https://programmesupport.pbworks.com/w/page/46353821/Supporting%20tools%20and%20frameworks

http://bit.ly/nfF8Mg

Support for baseliningDigital Literacy

Evaluation support- a starting point for conversations- DigLit programme wiki - follow up Elluminate session in November - project support Synthesis support- mapping project objectives and success criteria to programme view- develop resources on Design Studio- work with professional bodies

Jay Dempster

Helen Beetham