DIANA CANO, ETSTOM MURDOCK, MOODLE ROOMS

HEIDI LARSON, EDC

Moving On-Line Assessments Forward using an Open

Source Technology Platform

Findings from a study conducted byEducation Development Center

& Grunwald Associates LLC

Sponsored by ETS

Purpose of Study

• To find out: – Experiences with & attitudes toward

• What:– Internet-based testing and the potential for

an open source delivery platform• To help:

– Further the conversation on how technology can enhance and support education goals.

Study Participants

• Interviews with state assessment and/or ed tech leaders from 27 states

• 38 interviews with national education opinion leaders representing both public and private organizations

“Open source delivery platform” means, for this talk:

• A technology platform for the delivery of online assessments, accessible by signing a free licensing agreement, with public collaboration on software development and improvement. Vendor support may be available for purchase.

• It does not include:– Open content or Curriculum– Teacher-developed tests– Computers or other devices– Freeware

5

C States (11)9 Assessment 7 Technology

C States (11)9 Assessment 7 Technology

B States (11)10 Assessment

10 Technology

B States (11)10 Assessment

10 Technology

A States (5)5 Assessment Leaders

2 Technology

A States (5)5 Assessment Leaders

2 Technology

Defining Experience Distinctions: A, B, C States

Paper & PencilAccountability Testing

Statewide Online Accountability

Testing

Piloting and Experimenting with Online Accountability

Testing

Open Source Platform

Accountability Testing

Overall, a positive reaction

• Both State and Opinion Leaders are interested in learning more about an internet-based, open source assessment delivery platform.– Generally more favorable responses from:

• States with greater experience with online assessment (A & B States)

• Technology and Opinion leaders

• However, moving to an open source platform would be a large step forward: – Experience and knowledge of open source is

limited. (Overall, although C states in particular)– Many states have yet to adopt any significant

online testing, for various reasons.

7

Perceived benefits of an open source delivery platform

• Flexibility to customize to state requirements

• Adaptability of the platform (mostly Opinion and State Technology Directors)

• Community: Ability to share, build upon and transfer innovations across states

8

“It’s an effective use of resources in the sense that if there is an open source large scale assessment platform, then states could pool their resources together similarly as districts have to improve software around Moodle and some of the other open source technologies...” [Opinion Leader]

“It’s an effective use of resources in the sense that if there is an open source large scale assessment platform, then states could pool their resources together similarly as districts have to improve software around Moodle and some of the other open source technologies...” [Opinion Leader]

Benefits, and potential benefits,of an Internet-based platform

Nearly all of the key benefits mentioned were related to IBT:

9

Faster score reporting For constructed response items in particular

Logistics relief No more tractor-trailer trucks unloading boxes of testing materials

Fewer errors, greater integration Demographic data can be pulled in from a district database

Potential for interactive and multimedia items

To gain a clearer picture of what students know and can do, and where they are getting off track

Accommodating students with special needs

Adaptability of platform would enable more accessible assessment for all students

Students like the computer tests No report of students wanting to return to paper and pencil.

Other “potential benefits” of open source platform not embraced

• Cost savings: Respondents liked the potential of cost savings, but anticipated a greater burden on staff, infrastructure, and other resources.

• Vendor Independence: Often vendors were seen as a partner or at least as necessary and reliable support.

10

Primary concerns about anopen source delivery platform include:

11

Security Respondents not comfortable with perception of open source and Internet safety, at least not for high-stakes tests.

Infrastructure Availability of computers and servers, Internet reliability, bandwidth capacity

Expertise Lack of enough expertise at the state, district,and school level to support development and delivery

Going solo Generally, collaborative relationships are seen as desirable, if not necessary, by some for such an initiative to be successful.

Potentials of Collaboration?Participants Unsure

• Many respondents, in particular those familiar with open source programs and with consortiums, could see opportunities for collaboration:– Sharing the development and maintenance burden– Sharing quick-turnaround improvements from one state to

the next• Others were wary:

– No experience, or negative experience, with multi-state collaboration

– Concerned with having to compromise standards• Leadership and/or structured governance

– Considered key by many respondents

12

A StatesPerceptions of Feasibility

•Perceive concrete benefits of internet based testing

•Some respondents perceive open source solutions as a strong contribution to Common Core standards.

B StatesPerceptions of Feasibility

•Many are interested in potential benefits of an open source platform for high stakes testing, but would want to learn more or see results in other states before adopting an open source platform in their own state.

•Some saw ways that open source could be used in developing and sharing test items, particularly in relation to the development of Common Core Standards across states.

C StatesPerceptions of Feasibility

• Using open source would require major infrastructure improvement; many C state respondents do not see their state in a position to make this move.

Feasibility of an Open Source, I-Based, Delivery Platform: A, B, C States

Desired features in an open source platform

• Able to handle more complex item types to take full advantage of the technology

• User-friendly, intuitive interface with easy end user support tools

• Item development standards to allow greater sharing across formative tests, states and vendors

• Enhanced data aggregation and analysis tools

• Flexible reporting options

14

Strategies for SuccessFrom the public report: State officials experienced with internet-based assessments offered the following strategies to ensure successful implementations:

Start small: Stagger implementation of online assessments1.Start simple: Begin with multiple choice test items before venturing on to more complex items2.Stretch the testing window to allow all students access to the test3.Collaborate: The states that reported the most success with their online initiatives also reported strong working relationships among both assessment and technology experts.4.Safeguard: Include redundancies, checks, and safeguards to prevent data loss or corruption

To read more about this study: http://www.grunwald.com/pdfs/

Grunwald_Open_Source_Public_Report_v3.pdf

Or contact:

Heidi Larson (hlarson@edc.org) and Bob Spielvogel (bspielvogel@edc.org)

of Education Development Center

Or Peter Grunwald (peter@grunwald.com)of Grunwald Associates LLC

16

1

How open-source LMS became adoptedby classroom teachers

• Easier room logistics• Easier time logistics• Rich reporting on student activity• Rich reporting on instructor activity• Collaborative events• “Things you can’t do in a brick-and-mortar classroom”

1

How open-source LMS became adoptedby institutions

• Use of a common platform• Functionality built across the platform• Interoperability between upstream and downstream

systems• Collecting and acting upon longitudinal data• Understanding transitions• The transparency of open-source systems• Adoption Levels• Open-source and closed-source hybrids

2

Use of a common platform

Higher Ed Market

Economics

Legacy System Attrition

Growth in for-profit online

schools

Open-sourceAdoption &

Choice

Pluggable Architecture with

vendors

Market Seismic Shift

3

Functionality built across the platform

• Provide a platform that delivers:• Flexible deployment• Comprehensive features• Configurable Implementations• Industry-leading reliability• Security through transparency• High-quality, community-based development

4

Interoperability between upstream and downstream systems

• Open standards• IMS LTI• SCORM

• Pluggable architecture• More than 620 non-core modules

• Open API• Web-services

7

Secondary to Secondary to Post SecondaryPost Secondary

Post Secondary Post Secondary to Workforceto Workforce

2- 4 Year 2- 4 Year Higher EdHigher Ed

Academic Academic LifecycleLifecycle

Collecting and Acting upon Longitudinal Data

A growing number of states have policies to ensure High School Students graduate College and are workforce ready

Achieve.org Survey/Research 2007

Understanding transitions by tracking standards

Many States Are Aligning College- and Work-Ready High School Standards

9

The transparency of open source systems

• Core code is available to anyone• No vendor lock• Schools do not lose IP• Multiple vendors can write to open standards, open code

5

Adoption Levels

K.C. Green, Campus Computing, 2009

8

Open-source and closed-source system hybrids

Increase Capabilities

Increase Capabilities

• Focus internal resources on value-

added, strategic activities

• Transition to SLA-driven 24x7x365

end-to-end support models

• Leverage cloud modularity to enable rapid introduction of

new SaaS applications

• Focus internal resources on value-

added, strategic activities

• Transition to SLA-driven 24x7x365

end-to-end support models

• Leverage cloud modularity to enable rapid introduction of

new SaaS applications

Mitigate RisksMitigate Risks

• Eliminate technology and vendor lock-in

• Ensure solution scalability from pilot

projects through enterprise

deployments

• Enable cost effective disaster recovery

and business continuity solutions

• SaaS vendors can provide a wall that

protects proprietary elements of a

solution

• Eliminate technology and vendor lock-in

• Ensure solution scalability from pilot

projects through enterprise

deployments

• Enable cost effective disaster recovery

and business continuity solutions

• SaaS vendors can provide a wall that

protects proprietary elements of a

solution

10

Watching for the adoption of open source for assessment

Open Source is now mainstream.

“Adoption of open-source software (OSS) is becoming

pervasive, with 85% of companies surveyed currently using

OSS in their enterprises and the remaining 15% expecting to in

the next 12 months.” Gartner, Inc., “User Survey Analysis: Open-Source Software, Worldwide, 2008”

®

Using Open Source Software for Innovative Assessments

June 2010

®

What are “innovative” assessments?

• Exercise types that allow us to expand measurement beyond constructs possible with largely multiple-choice testing

– These include open-ended and performance testing

– They also include using technology to expand constructs• To test things we wished we could test on paper• To test skills and knowledge that do not exist absent the

technology itself

• Use of these items in high-stakes assessments

32

®

Delivering by computer has operational and measurement advantages

Item level computer adaptive test• Reduced testing time required for examinees• Greatly increased the number of items that must be authored, pretested,

scaled, etc.• Unique aspects of the automated item selection algorithm can be exploited

Item types are becoming more complex and technology rich• Test development and software development are becoming more closely tied• Opportunity for IT to directly impact assessment

33

®

Challenge #1: Cost

• Some of these sorts of items can be expensive and time-consuming to develop

– Particularly a problem for exercises for which we lack a set of “operational norms and practices” (e.g. simulations)

• Many of these exercises require human scoring, which adds cost

– Unlike development or analysis costs, these go up on a per-student basis

– Also add schedule time

– There are electronic scoring options, although these carry limitations as well

• Computer-based testing can add hidden costs (machines, development of larger item pools)even if tests use traditional items

– School computer labs need to keep pace and offer access for test takers

34

GN

U F

ree

Do

cum

enta

tio

n L

ice

nse

fro

mW

ikim

ed

ia C

om

mo

ns

®

Challenge #2: Test development know-how

• For some item types, little “operational knowledge” or templates for development exist in the industry

– We understand how to produce large numbers of multiple-choice items with known performance characteristics

– This knowledge also exists for some types of constructed-response items, although the cost of a mistake in development is far higher

• Need improved cognitive models since we cannot rely on the brute force of multiple-choice tests

35

Th

e T

hin

ker

, by

Au

gu

ste

Ro

din

th

e C

alif

orn

ia P

ala

ce

of

the

Le

gio

n o

f H

on

or

by

Ka

rora

; p

ub

lic d

om

ain

®

36

• Some types of items need computers and possibly longer windows

• We need to ensure that interfaces and activities are not so complex that it takes weeks for someone to learn to take the test

• Advanced item types may yield different group patterns, which may be an equity issue depending on construct definition

• We must make sure we remember it is an assessment

Challenge #3: Technology

®

Working with open source software for delivering assessments

• ETS’ Cognitively-Based Assessments of, for, and as Learning (CBAL) research initiative will use Moodle this year

– handling the tasks and activities associated with that research project

– integrating a proprietary testing engine for the assessment delivery

• OECD’s Programme for the International Assessment of Adult Competencies (PIAAC) uses Testing Assisté par Ordinateur (TAO)

– interviewing adults aged 16-65 years in their homes – 5,000 in each participating country

– assessing their literacy and numeracy skills and their ability to solve problems in technology-rich environments

37

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.

FOR MORE INFORMATION ON THIS OPEN SOURCE STUDY: HTTP://GO.EDC.ORG/OPENSOURCEPLATFORMREPORT

38

HEIDI LARSON (HLARSON@EDC.ORG)BOB SPIELVOGEL (BSPIELVOGEL@EDC.ORG) PETER

GRUNWALD (PETER@GRUNWALD.COM)TOM MURDOCK (TOM@MOODLEROOMS.COM)

DIANA CANO (DCANO@ETS.ORG)