The Need for Evidence Innovation in Educational Technology Evaluation

http://cede.lboro.ac.uk

Melanie KingHead, The Centre for Engineering & Design Education

The Need for Evidence Innovation in Educational Technology Evaluation

INSPIRE: International conference for Process Improvement, Research and EducationApril 2014, Southampton Solent University

The Centre for Engineering and Design Education

• The Centre for Engineering and Design Education

• The problem: The need for evidence innovation

• The 5 contributing factors that lead to narrow evaluation

• Recommendations: Realist Evaluation

Outline


The Centre for Engineering and Design Education (CEDE)

CEDE’s team of specialists work closely with the Engineering and Design Schools at Loughborough University to encourage effective practice and innovation in teaching and learning.

www.kit-catalogue.com

www.webpaproject.com

www.co-tutor.co.uk

http://www.kit-catalogue.com


The problem: the need for evidence innovation

src: http://sociologicalimagination.org

“we need to be at the forefront of the edtech revolution”

David Willetts MP

2011 marked a rapid and critical demise in funding = tighter budgets and smaller teams

How can we provide supporting evidence and make a case that in-house educational technology development is indeed a catalyst for innovation in T&L, and therefore justify further investment?

< 2011

2014 >


5 Factors that impact on effective edtech evaluation

Premature timing Inappropriate models Corporatization of HE

Complexity of agileLack of shared terminology


1. Premature Timing

Summative evaluations carried out immediately after an edtech development will never fully give an understanding of the potential influence and impact.

Learning tech development project

start end

IMPACT?

evaluationPROCESSOUTCOME

PRODUCT


IMPACT

src: http://en.wikipedia.org

CETL evaluation?

IMPACT


2. Inappropriate existing software techniques and models

Existing maturity models do not help us to fully understand the organisational factors that affect the potential for success of in-house edtech development.

Organization maturity

Users’ acceptance

Prediction of potential to nurture and

support edtech

SESRMM

CMM

eMM

Existing acceptance models do not help us to understand staff and students’

beliefs, attitudes and intentions with regards to adopting new edtech

UTAUT

[The UTAUT contributes to the study of technology adoption] “reaching a stage of chaos.”R. P., Bagozzi, 2007


3. Political context and the corporatisation of Higher Education

Higher Education is in such a rapid state of change that it makes contextual evaluations problematic with political drivers calling for quantifiable evidence of cost savings and efficiency


4. Complexity, agile development & participatory design

Homegrown edtech development is a complex and chaotic cycle of process and product improvement.

src: http://ciosp3.us/is-agile-development-good-for-health-it/

Students

Developer

Chaos is the science of surprises


5. Terminology – the semantics of edtech

The use of inconsistent terminology within the sector is a barrier to effective evaluation.

src: http://blogs.cetis.ac.uk

src: http://infteam.jiscinvolve.org

“Future research would benefit from a greater

degree of consensus over the use of common and

explicitly defined terminology.”

McLeod & MacDonnell, 2011

E-learning? Learning technology? E-administration? Educational technology? Edtech? Technology enhanced learning?


RECOMMENDATIONS

“What type of software works, in which conditions and for whom?”

“Did the software work?”

Pawson, 2013


RECOMMENDATIONS

src: https://www.wageningenur.nl


RECOMMENDATIONS

“Based on specific theories, realist evaluation provides an alternative lens to empiricist evaluation techniques for the study and understanding of programmes and policies. This technique assumes that knowledge is a social and historical product, thus the social and political context as well as theoretical mechanisms, need consideration in analysis of programme or policy effectiveness.” Pawson, 2013


RECOMMENDATIONS

Mechanism: what independent mechanisms, linked to edtech initiatives, may lead to particular outcomes in a given context?

Context: what complex conditions are needed for an edtech initiative to trigger mechanisms which produce particular outcomes patterns?

Outcomes pattern: what are the practical effects produced by causal mechanisms being triggered in a given context?

http://cede.lboro.ac.uk

Melanie KingHead, The Centre for Engineering & Design Education

[email protected]

INSPIRE: International conference for Process Improvement, Research and EducationApril 2014, Southampton Solent University

Prof Ray DawsonDepartment of Computer Science, Loughborough University

Dr Firat BatmazDepartment of Computer Science, Loughborough University

Prof Steve RothbergSchool of Mechanical and Manufacturing Engineering, Loughborough University

The Need for Evidence Innovation in Educational Technology Evaluation

Education

Transcript of The Need for Evidence Innovation in Educational Technology Evaluation