TAO

Open Source Solution For Successful Assessment Programs

What Is Open Source Software?• Software where the copyright holder makes the source code

available to anyone to modify and re-distribute it for any purpose

• Some most successful products that you may have heard of…

Security of OSS

• Does the DoD use OSS for security functions?YES.

• Doesn't hiding source code automatically make software more secure?NO.

• Since open source code is publicly released, anyone can review it, including for the possibility of malicious code.

Source: http://dodcio.defense.gov/OpenSourceSoftwareFAQ.aspx#OSS_and_Security.2FSoftware_Assurance.2FSystem_Assurance.2FSupply_Chain_Risk_Management

Bottom Line

• Open source software is of highest quality

• Open source enables economies of scale that are not available to proprietary products

• Assessment professionals benefit by being able to deliver highly innovative tests at a fraction of today’s cost

Brief History

• TAO initiated by University of Luxembourg and LIST

• Commercial services since 2012 through spin-off company O.A.T.

• Global community of contributors• IMS industry leadership

5

Fast Growing User Base

6

With TAO You Can

1. Author your own content2. Ingest 3rd party content3. Deliver tests in the cloud4. …or from your notebook5. Manage students6. Create reports

7

Innovation At Its Core

• Align your content to any standard

• Linear scalability thanks to NoSQL delivery engine

• Design immersive test experiences that are truly portable

Why Users Select TAO

• It’s innovative

• It’s completely open

• Cuts assessment cost by 50% or more

9

Open Source K12 AssessmentsA Roadmap for the Future

12

1. Vision2. What’s been done so far3. Review Open Source Assessment Platform Components4. An overview of the current state of the Open Source Assets5. What is not part of these platforms?6. What will happen next?7. How do we get there from here?

OPEN SOURCE ASSESSMENT ROADMAP

13

VISION

The entire world is leveraging open source software in all aspects of business and technology.

We believe we can core components for assessment construction and delivery, and that we can leverage

scoring, reporting, administrative interfaces that may be common within the K12 industry.

We believe this has the potential to fundamentally improve the economics and features of assessment.

14

TRENDS

What’s driving us to believe these things?

15

TRENDS - STANDARDS

IMS – QTI and LTIQTI started slow, but it is now ubiquitous. LTI has been

adopted by Over 130 edtech vendors and platforms!

16

TRENDS – A UBIQUITOUS INTERNET

• $5.5 Billion in federal and private sector investment

• Goal is to have 99% of students on broadband in the classroom by 2018

17

TRENDS - REUSE

After all how many rostering, reporting, content management and delivery systems do we need to

support large-scale assessments?

There are two models of reuse – license a proprietary platform and leverage an open source platform.

We are realizing that there are better models of reuse than proprietary software licensing. And that these

models have been validated by large scale assessment implementations in the U.S. and internationally.

Open Source K12 Assessments

19

SUMMATIVE ASSESSMENT - NCSC CONSORTIUM

• 2 pilots in 2014; Operational test in 2015

• Participating States:

AR,AZ,CT,DC,ID,IN,ME,MT,NM,RI,SC,SD,CNMI,GUAM,VI• States included in Pilots, but not Operational test:

CA,MD,FL,MN,ND,PA,TN,VA

• 1,342 unique items

• 64 test forms

• 2,619,290 responses

• 74,040 assigned tests

20

SUMMATIVE ASSESSMENT - NCSC CONSORTIUM

What did we learn?

• Completed or cancelled tests: 96.1%

• 2,172 Measured Progress help desk calls• About 10% resulted in a data change or a possible software defect (43)

• It was possible to switch testing vendors between pilot and

operational testing with minimal disruption

• Areas identified for improvement in the platform include:• Audit history, backend queries for CSR, System Reporting, Data

changes for CSR

21

NON-SUMMATIVE ASSESSMENT – PARCC CONSORTIUM

• Emphasis on diagnostic and lower-stakes assessment delivery

• Extended with adaptive item selection; integrating Pearson item

selection algorithm

• Integrating items developed by other vendors

• New item types developed against PARCC performance objectives

• New test driver designed for disconnected delivery

• Student tools, accessibility

• PCI Architecture

• Still in development – launch in 2015-2016 school year

22

NON-SUMMATIVE ASSESSMENT – PARCC CONSORTIUM

What are we learning?

• Systems integration and content sharing are challenging to program

productivity

• SSO is expensive to get right and standardize on across platforms

• Different test programs have large differences in platform

requirements and impact on user experience• Interim reports • Dynamic test assignment and the ability to personalize tests and align

to specific common core standards is essential for non-summative assessments

• The ability assess across grade levels is also important

K12 Open Source Assets

US K12 OPEN SOURCE ASSETS

24

US K12 NCSC OPEN SOURCE ASSETS

25

26

• Detailed classroom and school reporting: This is being developed currently under a separate support contract with another testing vendor;

• Human scoring: Allowing teachers or scoring teams to log in and provide grading against rubrics as they review student responses is still in development;

• Out-of-the-box Adaptive Delivery: Staged-adaptive section selection was deferred from the year one operational test. R&D was completed and shown that it could be implemented with small extensions based upon features available in QTI.

ELEMENTS THAT ARE NOT AVAILABLE

US K12 PARCC OPEN SOURCE ASSETS

27

US K12 OPEN SOURCE ASSETS

28

29

• Detailed classroom and school reporting: This is being developed currently under a separate RFP with PARCC by Amplify. It may be available to the community in 2017.

• Human scoring: Allowing teachers or scoring teams to log in and provide grading with rubrics as they review student responses is still in development

• Out-of-the-box Adaptive Delivery: Adaptive item selection still requires integration with a vendor’s library

• Smarter Balanced Item Ingestion and adaptive algorithm integration• Next Generation Science Item types and tools• High Stakes security features and disconnected delivery testing

ELEMENTS THAT STILL REQUIRE DEVELOPMENT

An Open Source Ecosystemfor K-12 Assessments

OPEN SOURCE ECOSYSTEM

Technology

Holding Company

Community

Services Components

Customizations

Users – States, districts, publishers

FreeLicensePartner

Partner

Partner Network

How We Get There: Roadmap

33

US K-12 ROADMAP – WHAT’S NEXT

There are a variety of ways that we are going to extend this platform in the future:

1. Seek grant funding to continue development;2. Implement the platform and extend it with assessment revenues for

consortia, states, and districts;3. Breakthrough and O.A.T. will make investments to continue to lower the

cost of deployments and extend the feature set;4. Partner with O.A.T. to build out features that are critical for the K-12

market;5. Partner with test publishers edtech developers, and state IT organizations

who desire expertise on implementing, integrating, and extending the platform.

34

SPECIFICALLY…

1. Continue to deploy PARCC’s year one nonsummative product;2. Help all NCSC member states and/or their test vendors deploy the NCSC

platform for their 2015-2016 alternate assessment;3. Seek grant funding to build out some missing features and complete the

work required to demonstrate the idea a single, unified assessment platform;

4. Demonstrate the platform’s ability to deliver SmarterBalanced Assessments;

5. Continue to work in and out of the K-12 market to complete missing assessment features and continue the evolution of the TAO platform;

6. Develop certification, training, and services to allow states and districts to utilize the platform and broaden the resource pool who can deploy, integrate and extend it.

35

WHAT WE’RE REALLY DOING

We want to fully realize the investment from the RTTA to create an enduring assessment platform ecosystem that the U.S K-12 Assessment community can deploy today, extend

tomorrow, and continue to utilize for decades to come.

This is bigger than one person, one company, one state, or one consortium’s vision.

Thank you!

Doug Wilson(847) 864-0033 x201doug.wilson@breaktech.com

National Center and State Collaborative Open Source Assessment Solution

CCSSO National Conference on Student AssessmentJune 2015

Chris DomaleskiCenter for Assessment

Overview

• NCSC is a consortium of 24 states along with University and non-profit expert partners, working together over four years to develop coordinated curriculum, instruction, and assessment resources for students with the most significant cognitive disabilities.

• In spring 2015, 14 states participated in successful operational testing, which involved over 70,000 tests administered to over 35,000 students.

38

NCSC Assessment System • The NCSC Assessment System is a technology platform that facilitates a

range of functions to include– Assessment creation, management, and delivery

– Evidence collection and scoring

– Reporting

– Training resources

– Data management

• NCSC partners contributing to the development of the solution include: CTB McGraw Hill, Measured Progress, Open Assessment Technologies, and Breakthough Technologies

NCSC Assessment System

40

Operational Open Source Solution

• What does it mean that the NCSC System is Open Source?– Virtually all of the system is built with software in which the source code is made available and

licensed such that the rights to study, change, and distribute the software are freely available to anyone.

– NCSC will publicly release the code this summer

• What benefits does this provide?– Cost Reduction: No licensing fees

– Transparency: The software code is completely available for inspection and scrutiny

– Flexibility: Because the code is freely available, it can be easily adapted, revised, or expanded as needs warrant.

– Collaboration and Support: Everyone benefits from the cumulative and collective improvements efforts.

System Enhancements

• The enhancements to the system developed during the project include– Creation of a system to manage users and organizations and track progress

– Assessment features for user accessibility

– Evidence upload functionality to capture and upload student work

– Presentation of content on printable PDFs for paper-based administration;

– Improvement and expansion of delivery and navigation tools

42

Post Implementation ‘Reflections’

• The system worked without disruption or malfunctions

• Feedback on enhanced assessment features and user interface improvements was positive

• ‘Administration’ functions continue to be the most prevalent areas requiring user support

• Looking ahead, NCSC seeks to identify efforts/ resources to support the open source community

43

Additional Resources

• www.ncscpartners.org

• Technology architecture and specifications

• Presentations, papers, handouts and more for various audiences.

Chris Domaleski

cdomaleski@nciea.org

The Progress and Promise of Open Source Technology to Support

State Assessment Systems

June 23, 2015

• The Partnership for Assessment of Readiness for College and Careers (PARCC) is a consortium of states working together to deliver a set of assessments that measure whether students are on track to be successful in college and their careers.

• Summative Administration (2014-15)– Over 15 million tests completed online (PBA and EOY combined)– Several days with over 1 million students testing – Our busiest day: 1.4 MILLION students; 204K students in an hour– 100% uptime– States reported higher-than-expected online testing rates, some with more

than 90% of students test online

PARCC

46

• Two pronged assessment approach– Summative assessment – Partnership with Pearson

o Pearson Access Next– System management

o Pearson TestNav (v8.2 first Cloud based solution)– Test administration

– Non-summative assessments, instructional tools & professional development – develop PARCC Platform (Open Source)o Assessment Delivery System (ADS)o Partnership Resource Center (PRC)o Data Management & Reporting System (DMR)

– Production release – Fall 2015

PARCC

47

PARCC Platform Solution Diagram

48

PearsonSummative Assessment

Platform

Data Management& Reporting (DM&R)

IndividualSummativeTest Results

BreakthroughNon-Summative Assessment

Platform (ADS)

IndividualNon-Summative

Test Results(Diagnostics)

BreakthroughPartner Resource Center

Platform (PRC)

IndividualNon-Summative

Test Results(Speaking & Listening/K-2)

End-UserValidation

End-UserValidation

Individual Score ReportBatch Generation > Schools

Online ReportingStates/Districts/Schools

Diagnostics & Instructional Tools

Summative &Non-summative

Reporting

"Linus' Law," named for Linus Torvalds, the creator of Linux. Given enough eyeballs, all bugs are shallow.

What that means is that the more people who can see and test a set of code, the more likely flaws will be caught and fixed quickly.

The opposite of the security through obscurity argument (secrecy of the design or implementation to provide security)

to justify the use of expensive proprietary products.

Open Source

49

My first time.

The Benefit of Open Source

50

• Value proposition for open-source solutions– No license fee

o Commercial costs of software licensingo PARCC mission: Drive the costs of assessment as close to zero as

possible– Customizable

o Commercial products with a generalized feature/function set for mass audience

o Allows for “easier” pathway for integration and enhanced functionalityo Often much more standards based – “Interoperability”

– Deep developer support network– Community benefits

The Benefit of Open Source

51

The Benefit of Open Source

52

• Open Source does not mean free– Development costs

o Platform developmento Content development

– Technical operations costs– Operational support costs

• Release strategy– Do we encourage States to adopt a common platform or

leverage the released code to build out a customized solution

Issues implementing Open Source

53

• All Open Source is not equal– Make sure you are working with reputable resources that can

support your efforts (BT & OAT)• Be flexible, your plans will change, but be diligent on scope• Collaborate… Talk to others, talk to us

Jeffrey CuffDirector of Technologyjcuff@parcconline.org

Advice

54

Going Open-Source: Learning from the Pioneers

Michael RussellBoston College

Key Lessons• Open Source can support:

– Assessment program control over enhancements on their timelines

– Use of the same platform across different operational vendors

– Separation of content development from content delivery

Key Lessons• Open Source assessment solutions may require:

– Integration of multiple open source solutions – Connections with in-state systems– Adoption/Refinement of industry-recognized

content standards– Clearly articulated strategy for contributing code

back to the community

Key Lessons• Open-source can work

– But look for:• Mature solutions• Established community• Tested solutions

– And be ready to:• Start early• Invest in customization• Partner with experienced developers• Pay for installation, hosting, and routine maintenance• Control your delivery platform

Looking to the Future• Can TAO or other solutions be generalized to meet a

higher percentage of state assessment program needs “out of the box”?

• Can TAO or other solutions be used to support multiple assessment programs?

• Can enhancements made by individual programs be coordinated to reduce redundant efforts and gain greater efficiency?