Evidence-Centered Approach to Online Assessment of Students’

Digital Competence

Mart Laanpere, Kai Pata (Tallinn University)Mario Mäeots , Leo Siiman (University of Tartu)

Problem space• Need to assess digital competence: a newly

defined key competences in Estonian national curriculum for primary and secondary schools

• Requirements: – Compliant with DigComp framework– Implemented online using e-exam system EIS – Scalability– Automated assessment– Reliability, validity

Digital competence

• Three competing approaches to digital competence/competencies:– Theory-based knowledge, derived from the academic

discipline of computer science (computational thinking)– Pragmatic skills, needed at the workplace (ECDL)– Key competence, contextualised, closer to “learning to

learn” skill• DigComp follows the second approach, Estonian

curriculum combines 2nd and 3rd

Teaching and assessing ICT skills in Estonia

• 1986: Computer science as a compulsory theoretical subject, no exam, teacher-created tests

• 1996: Informatics as an elective subject, learning generic ICT skills for office jobs

• 2000: ICT and media as cross-curricular theme, pilot exam on national level (test + practical task)

• 2011: informatics as an elective subject, digital competence as a key competence

Research design and organisation

• Two teams of researchers from two different universities (University of Tartu & Tallinn University)

• Combination of two approaches: – Inductive/descriptive, bottom-up (Tartu team):

collecting and analysing the best practice from large-scale comparative and national studies

– Deductive/prescriptive, top-own (Tallinn team): building a new framework from scratch, deducing the new type of practice

ECD framework (Almond, Mislevy et al. 2015) inspired by e-exams of Cisco Network Academy;Has been applied in various contexts: Assessing languagecompetence,Inquiry skills in STEMEtc.

Layered assessment architecture of ECDLayer Role Key entities Examples

Domain analysis

Collect info about domain, teaching practice, frames

Concepts; terminology; tools; representation forms

Curricula, rubrics, test

Domain modelling

Express assessmentargument in narrative form

KSAs; potential work products;potential observations

Design patterns

Assessment framework CAF

Express assessmentargument as blueprints for tasks or items

Student, evidence, and task models; student-model, rubrics etc

Task templates

Imple-mentation

Implement assessment,presenting tasks or items and gathering andanalyzing responses

Task materials (incl.all materials, tools, affordances); work products

Rendering protocols for tasks; IMS/QTIrepresentation of materials & scores;

Delivery Coordinate interactions of students and tasks; task and test-level scoring; reporting.

Tasks as presented; workproducts as created; scores as evaluated.

summaries forindividual and group-level reports;

Assessment framework (CAF)

What?

How?

Where?

Extent?

Domain analysis: example

Performance taxonomy

Fact Concept Procedure Rule

Knows

Understands

Applies

HOTS

Based on: Merrill ja Feisel-Schmitz

Bayesian network

Information & Data Literacy

Digital content creation

Communication & collaboration

Digital safety

Problem solving

KSA (Knowledge-Skills-Abilities)P_Info P_Comm P_Content P_Safety P_Probl

T_Info

T_Comm

T_Content

T_Safety

T_Probl

Example: practical task

• Create a presentation that compares two countries: Estonia and Slovenia

• Criteria: – At least 4 slides/screens– Diagrams based on reliable and recent data– Images (IPR need to be followed, licensing)

Piloting

• Not completed, planned in 6-8 schools this October

• Bayesian data analysis, evidence accumulation• Scenario-based design of new task templates• Exploring the solutions for simulation tasks

(inspired by examples collected by Tartu group)

Questions for discussion

• Any experiences with ECD?• How to combine inductive and deductive

approaches?• Semi-automated assessment of practical

tasks?