Delivery platforms for national and international computer-based surveys

History, issues and current status

Sam HaldaneAustralian Council for Educational Research (ACER)

PISA CBAS

• Objectives– Comparability– translation– rich content

• Requirements– Security– Hardware, software– Test Administrator knowledge

CBAS - Implementation

• Installed client / server software• Java– Java Media Framework (movies and sounds)– Internationalization / localization support– Networking

• Hardware platform– Client / server carry-in laptops + network

CBAS - Economics

• Delivery – carry in laptops = high cost and low failure rate

• Data capture – data management procedures• Marking – not required due to nature of items• Ease of translation– Low word count– Must install custom software– Translation of interactive elements

PISA ERA

• Objectives– Lower cost– Logistically easier– Use existing school infrastructure– Comparability not so important– Complex hypertext environment

PISA ERA

• Requirements– Cannot assume inter / intranet– Cannot assume host OS version– Cannot always install software– Must run on lowest common denominator

hardware

PISA ERA

• Implementation– Bootable CD with USB storage in field trial– Bootable USB option in main survey– Linux OS using TAO as delivery platform– CD / USB contains all software needed (server,

browser, Flash etc.)– Minimum possible hardware requirements– Need for TA to change boot order in BIOS

ERA - Economics

• Delivery – using existing school infrastructure = low cost but high failure

• Data capture – de-centralised on USB storage – data management logistics

• Marking – expert marked items, need for online marking software

• Translation – need for separate translation system, and localization of marking system

NAP – ICT Literacy Assessment• Commissioned by MCEETYA1

• Australia wide assessment of ICT literacy in year 6 and year 10 (12 and 16 years old)

• Measures ability to– use ICT appropriately to access, manage, integrate

and evaluate information– develop new understandings– communicate with others in order to participate

effectively in society1 Ministerial Council on Education, Employment, Training and Youth Affairs

NAP – ICT Literacy Assessment• Three deployment models– Internet delivery – school computers used as

clients with remote server– Carry-in server – school computers used as clients,

with server laptop plugged into school LAN– Carry-in minilabs (10 machines) – same model as

CBAS, for less equipped schools

• High success rate - 99% school-level response rate after replacements

NAP – ICT Literacy Assessment• Technical issues– Remote desktop client requires non-standard port– Internet Explorer plugin requires installation, and

Windows OS– TA training for all three deployment models– Schools requiring minilabs usually remote

CBAS vs ERA• Carry-in laptops vs bootable CD / USB

• Comparability issues – ERA hardware differs greatly whereas CBAS was standard

• High failure rate in ERA, perhaps more so in some schools

The future

• Toolbox of delivery methods that are used depending on school infrastructure

• Internet delivery will have highest ROI – ease of deployment, administration costs, but lack of infrastructure

The future

• No silver bullet• Carry-in server model best trade off at the

moment• Remote desktop based solution (for some

projects)• Interactive content with Flash• Ajax – Google Apps possible future direction