www.iktsenteret.no

Ove Edvard Hatlevik and Ingrid Radtke

An Application of Standard-Setting Methods in a Formative Assessment in Digital Responsibility among Norwegian Eight Graders.

Agenda

• Introduction

• Digital skills in the curriculum

• Digital responsibility

• Experience with standardsetting

Norwegian Centre for ICT in education

• Governmental agency

• Activities– Online guidelines for schools and educators– Summarizing experiences with methods and tools

in the classroom– Surveys and monitoring

Assessment

• Summative assessment

• Formative assessment– For example voluntary learning supportive tests– Feedback on how to improve

What to measure?

• Concept– Digital competence or ICT literacy or …

• Framework– E.g. the DIGCOMP framework (Ferrari, 2013) has

five areas with 21 dimensions and three proficiency levels

– European Commission, Joint Research Centre, Institute for Prospective Technological Studies

Different concepts

• ICT literacy (ETS, 2006; Ainley et al., 2007)– Computer and Information literacy (Fraillon et al.,

2013)– The International Computer and Information Literacy Study

(ICILS) 2013

• Digital literacy (Aviram & Eshet-Alkalay, 2006 )• Digital competence (Erstad, 2006; Ferrari, 2012)• Internet skills (Van Deursen & Van Dijk, 2009 )• iSkills (ETS, 2008)

About the concepts

• Often tools or technology (e.g. ICT/Digital/Internet) combined with learning domains (e.g. skill, competence, literacy)

• Definitions often include modes (critically or reflectively) or purposes (work or leisure)

One example of an international used definition of CIL

• Computer and information literacy refers to: “individual’s ability to use computers to investigate, create, and communicate in order to participate effectively at home, at school, in the workplace, and in society” (Fraillon et al., 2013, p. 17).

– The International Computer and Information Literacy Study (ICILS) 2013

International frameworks

• iSkills (Educational testing service, 2006)

• ATC21s (Binkley et al, 2012)

• DIGCOMP (Ferrari, 2013)

• Computer and information literacy (Fraillon et al, 2013)

• Australian Curriculum, Assessment and Reporting Authority (ACARA) (2012)

National initiatives to assess some ofthese concepts / frameworks

• Australia• Chile• Norway• Korea• U.S.

• International Computer and Information Literacy Study 2013 (Fraillon et al., 2014)

National Curriculum (Norway)

• Digital skills as a “key competence”

• Sources describing digital skills– A general framework of basic skills– A general description within the subjects– Explicit competence aims for the subjects• The Norwegian Directorate for Education and Training

Digital skills …

• … involve being able to use digital tools, media and resources efficiently and responsibly, […]

• … include developing digital judgement by acquiring knowledge and good strategies for the use of the Internet.

• The Norwegian Directorate for Education and Training (2012, p. 12)

General framework of digital skills

To acquire and process digital information

To produce digital information Digital judgement Communication

Example of a digital judgement task from the ICILS 2013

Example of description: Digital skills in Natural science involves

• … […] mastering searching and research strategies, learning to evaluate sources critically and selecting relevant information about themes within natural sciences.

Examples of one competence aims (social science):

• use digital tools to present work in the social sciences and comply with the rules for personal data protection and copyright.

Assessment team

• 7 employees at the ICT-senter

• Different skills and background

- teaching-experience- ict-skills

- research and statistics

- assessment skills

Assessment in Digital Responsibility

• One out of 5 formative assessments in digital competence

• In cooperation with the Directorate of Education

• Available from november 2015

Assessment development

• 3 rounds with piloting (and visiting schools)• Analyses with classical test-thoery and IRT• Standard setting with Angoff- and Bookmark-

method

Definition Digital Responsibility

• Digital judgement means being able to use digital tools, media and resources in a responsible manner, and being aware of rules for protecting privacy and ethical use of the Internet.

• Search and process: … interpret digital information appropriately and critically.

• Produce …. Be aware of copyright issues.

Digital Responsibility

• Items were developed in 6 content areas– critical use of digital sources– ownership of digital material– safety issues– personal integrity– cyber bullying– green data

Scoring and item-types

• scoring - each correct answered item gives one score.

• item-types - multiple choice and multiple response, knowledge matrix, matching, pull-down list and drag and drop.

Exsamples for Competence Aims

Norwegian• use different types of digital

and paper-based dictionaries

• be familiar with copyright rules on the use of sources

Application of Digital Responsibility• Use different types of digital

dictionaries

– critical use of digital sources– safety issues– personal integrity– ownership of digital material – cyber bullying– green data

Development of Assessment

• Autumn 2013, discussion and small pilot with 2 classes

• Spring 2014, 90 items with 1395 students from 36 schools

• Nov 2014, 60 items with 1062 students from 26 schools, representative sample

Final assessment – 50 items

Performance standards

• Three performance standards

• Low – middels- high• Lowest level was

divided into two

Performance standards

• Related to the items in the test

• Not the ‘ideal’ student Level 1 At this level, the pupil has basic skills in the area of digital judgment that encompasses understanding of basic concepts and risks of using digital tools. The pupil can make simple search for sources, have knowledge of TONE-term (method used to estimate credibility, objectivity, accuracy and suitability of digital sources) and can choose appropriate sources. When it comes to copyright pupils can be expected to have an understanding about what is right or wrong when referring to sources and downloading files. Privacy:

• Simple assertions about what is allowed to publish when it comes to photos or videos online • What kind of information pupils should provide about themselves and what dangers are associated with registration at various websites • Storing data and using search engines from their mobile phone • Simple concepts around privacy, like permission and consent

Safety: • Protect your own password when logging onto someone else’s PC • Dangers around using free wireless networks

Standard setting workshop

• 10 experts• Help from Cito to

conduct the workshop

Expert panel

• Teacher with hands on experience with the use of ICT in schools

• Other subject-matter experts• Stakeholders?

About the workshop

• Introduction of 3 performance standards• Training session with Angoff-method• Both Angoff- and Bookmark-method were

used

Angoff method

• Suppose a group of 100 borderline students. How many of them will answer this item correctly?

• Cut-off points. Sum judgements.

• Reflections

Bookmark method

• List with ordered items by difficulty after IRT-analyses

• Experts choose the items which mark the borderline for the levels of the students.

• Cut-off points: average of the bookmarks set by experts.

Results

• About the same cutting point for the higher level, level 2-3

• Higher cutting point for the lower level after Angoff-method, level 1-2

• Standard-deviation

Decision-making

• Which factors would be important?

1. Distribution of items and students

• Number of items to qualify for a certain level.• Number of students on that level. (As seen

from the pilot-data.)• Difficulty of items around the cutting-score.

2. More validity arguments

• Level 3 should be exclusive!• Not too many students in Level 1!

Guidance material

• Group-level• Item-level• Organisation-level

Way forward!

• Limitations?• Science versus individual judgements

• Validity – accept among teachers• Aim is to improve teaching and learning!