An Overview of the Higher Education ICT Literacy Assessment Ilene Rockman Gordon Smith California...

An Overview of the Higher Education

ICT Literacy Assessment

Ilene Rockman

Gordon Smith

California State University

Irvin R. Katz

Educational Testing Service

http://www.ets.org/ictliteracyCopyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

2Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.

Importance to Higher Education

• Measures cognitive problem solving abilities within technological environment

• Clarifies and articulates ICT student learning outcomes

• Provides evidence for accreditation purposes

• Simulates academic, work, or personal queries

• Engages students through critical reading, reflecting, and taking action


How the Assessment Differs From Others

• Interactive, web, and performance-based

• Real-world storylines

• Higher order reading and thinking skills

• Uses digital technologies, communication tools, and networks to solve cognitive problems in an ethical and legal way


Student Feedback fromBeta Tests• Never taken test like this before

• Challenging; took it seriously

• Interface was easy to use

• Required thinking and technical skills

• Enjoyed real-world storylines

• Tasks reflect activities at school, work, or home


Why Measure ICT Literacy?

• ICT is changing the very nature and value of knowledge and information and impacts the way we live, learn, and work

• ICT literacy is not simply a mastery of technical skills, but is the application of technical skills in an information society.

• There is a lack of information about the ICT literacy of students, and debate about how best to address this issue in academic curriculum


What is ICT Literacy?

Ability to use digital technology, communication tools, and/or networks to access, manage, integrate, evaluate, create, and communicate information ethically and legally in order to function in a knowledge society


Building on Prior Efforts

• ACRL Information Literacy Competency Standards for Higher Education

• ISTE National Educational Technology Standards

• Digital Transformation: A Framework for ICT Literacy (ETS, 2002)


ICT Literacy: a bridge between

Technical Literacy Information Literacy

DatabaseWord

ProcessingPresentation

• Can I bold a word?

• Can I open a database?

Access Evaluate Use

• Can you find information?

• Can you evaluate authority?

Information and Communication Technology Literacy

–Can I find information on the web?–Can I create a persuasive presentation?


Proficiency Model

ICT Literac

y

Define

Access

Manage

Integrate

Evaluate

Create

Communicate

Cognitive

Ethical

Technical


The ICT Literacy Project

• Two separate products– ICT Literacy Large-Scale Assessment (2005)

– ICT Literacy Individual Assessment (2006)

• Unique aspects of products:– Focus on cognitive skills rather than technical skills

– Simulation-based tasks – Not multiple-choice

– Generic software – Resistant to changes in technology

– Scalable and sustainable


– What new courses should be offered?

– Which courses should receive additional resources?

– Accreditation: Have our investments in ICT education been paying off?

– What courses should I take?

– In what majors will I be successful?

– Am I ready to enter the workforce or graduate school?

• Institutional comparisons across campuses or departments from year to year

• Individual decision-making

Potential Uses for Higher Education


Basic Design Features

• Interactive tasks using simulated software

• Delivered via the web in a 2-hour test

• Tasks of varying lengths (4-, 15-, and 30-min)

• Automatic scoring, with multiple scorable elements per task

• Evidence-based design

• Valid and reliable


Recent Test Administration(January – April ’05)

• Completed: 4582 examinees across 31 campuses

• Both Four-year and Two-year institutions

• Overall Reactions: “Interesting & fun, but exhausting with a lot of reading”

• Library community feedback:– Not enough emphasis on print media

– Too much emphasis on technology


Responding to the Feedback

• Investigate instances of “too much reading”– Identify problem tasks

– Reduce scenario descriptions

• Increase number of scenarios that incorporate print media (e.g., on-line library catalog)

• Keep same level of technology– Assessing cognitive skills

– Technology already minimal

– Accurately reflects disparate views of information formats that exist in the disciplines


Sample Task: Comparing Information

• Short task: Four minute duration

• Assesses one ICT Literacy subproficiency– Integrate: The ability to interpret and

represent information in an ICT context. This includes the ability to synthesize, summarize, compare, and contrast information from multiple sources.


Within the task

• Summarize information from three different types of sources

• Compare the information to reach a conclusion


In this task, examinees summarize information from a variety of sources and then draw conclusions from their summary.

Copyright © 2005 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service.


Examinees are presented with an information need and three different types of information sources:




1. Webpage



2. Email




3. Advertisement

The three sources present information according to different conventions. A successful candidate must be able to locate the relevant information in each source.




Deciding how to compare the sources involves identifying the requirements of the stated information need.



After filling in the table, examinees must interpret the summary to rank the three sources correctly.



Other Tasks

• Access: Simulated web or database searches

• Evaluate: Judge authority, currency, bias of search results

• Create/Communicate: Develop presentation slides based on accessed information


Web Search



Database Search



Database Results



Create a Presentation



Score Reports

• Test use guide and proficiency descriptions

• Interpretation guide

• Overall scores and four sub-scores– Define/Access, Evaluate, Manage/Integrate,

Create/Communicate

• Institution-specific contrasts


Group

Percentage of Students In Your School

Percentage of Students

Participating

n=25,000 n=112

Gender

Male 45 45

Female 55 55

Race/Ethnicity

Asian 10 8

Black 20 22

Hispanic 20 18

Native American 5 7

White 45 45

100 100


Current Higher Education Partners

• The California State University • Oklahoma State University • Portland State University• Purdue University• University of California, Los Angeles• University of Louisville• University of Memphis• University of Washington


Summary• Made for and by higher education

– Original 7 institutions spent a total of about 1000 hours reviewing, planning, and designing

• Scenario-based– Simulations, not multiple choice

• Psychometrically sound• Provide information for

– Accreditation & institutional improvement – Individual certification

• Dynamic and responsive

Questions & Discussion

Visit us at:

http://www.calstate.edu/LShttp://www.ets.org/ictliteracy


http://www.calstate.edu/LS



http://www.ets.org/ictliteracy

An Overview of the Higher Education ICT Literacy Assessment Ilene Rockman Gordon Smith California...

Documents

Transcript of An Overview of the Higher Education ICT Literacy Assessment Ilene Rockman Gordon Smith California...