Race To Accessibility:The Past, Present, and Future of Accessible Assessments 

The Council of Chief State School Officers

National Conference on Student Assessment

June 23, 2015

Panel of Presenters Trinell Bowman

Program Manager, Maryland State Department of Education PARCC –

Partnership for Assessment of Readiness for College and Careers

Jan McSorleyHead of Accessibility for School, Pearson

Wayne DickSenior Research Fellow, Knowbility, Inc.

Dave EdyburnProfessor, University of Wisconsin-Milwaukee

Linda ZimmermanSr. Assessment Solutions and Design Manager, Pearson Assessment Centre

Paul Grudnitski, in absentia TestNav Chief Architect, Pearson

The Evolution of Accessible Assessments

Drivers for Accessible AssessmentsThe reauthorization of IDEA in 1997 required that students with disabilities participate in state and district assessments. It also required that for students who could not participate in statewide assessments with accommodations that an alternate state assessment be developed.

By 2004-2005, more that half of the states requested test vendors to use universal design principles in developing their assessments.

Beginning in 2005, a few states begin developing online assessments which provided some additional opportunities and challenges for online testing.

Web Content Accessibility Guidelines 2.0 (WCAG)

Section 508 Amendments to the Rehabilitation Act; Usability Guidelines based on Section 508. http://guidelines.usability.gov/

Accessible Portable Item Protocol 1.0 (APIP)

1997 2001 2002-2004 2005 2006 2007 2008 2009 2010

IDEA and NCLB

A series of Enhanced Assessment Grants were awarded that focused on

accessibility and accommodations

Statewide Testing and the Shift from Paper to Online Testing

Universal Design

Applied to Assessment

Bias and Sensitivity Reviews

Test Available in Braille and Large Print,

etc. Race to the Top

Race to the Top Fund Assessment Program Selection Criteria

(A)(1) Governance - (up to 20 points) (A)(2) Theory of Action - (up to 5 points) (A)(3) Assessment Design - (up to 55 points) (A)(4) Assessment Development - (up to 35 points) (A)(5) Research and Evaluation - (up to 30 points) (A)(6) Professional Capacity and Outreach - (up to 15

points) (A)(7) Technology - (up to 10 points) (A)(8) Project Management - (up to 30 points)

PARCC Strategies for Increasing Student Access Provide item developers with clear guidelines for writing items that are free

of bias, are sensitive to diverse cultures, are stated clearly, and use consistent formats

Require item developers to use principles of Universal Design to allow participation of the widest possible range of students, and increase the likelihood that test questions measure only what they are intended to measure

Conduct bias and sensitivity reviews and statistical procedures that are designed to detect bias as part of the item development/field testing process

Develop common test accommodation and participation policies for SWDs and ELLs

Use technology to provide and increase access to testing accommodations

Conduct research to determine factors that promote or hinder accessibility

Accessibility Priorities for thePARCC Assessment System The Assessment System must be based on universal design principles and

other research-based principles such that all applications will support access for all students and staff.

The content (e.g., passages, items) must be associated with meta-data that describes any changes that will be made to the content, display, or input method necessary to provide appropriate accommodations support to the student.

The overall approach must leverage the use of computer-based accessibility tools, driven by an item tagging system that will control and ensure appropriate application of those tools at the item level.

PARCC included accessibility requirements as part of the PARCC Architectural Design and Specifications

Assessing Students With Disabilities

Goal: Assure that the examination measures what it is intended to measure, rather than reflecting the individual's impaired sensory, manual, or speaking skills. (American with Disabilities Act Title III, Part XII)

Challenges: Different technology used in class and on tests Lack of Braille access to tests Lack of AT and browser support for mathematics content New items types available – often relying on visual representations Providing access to instructional supports Security and compatibility with 3rd party technologies

Lessons Learned and Counting…. Incorporate the use of accessibility training during item

development and review trainings.

Engage students in the use of cognitive labs and usability studies

It is important that the item types are accessible and do not create further barriers The use of scrolling The number of items on a screen Use of technology enhanced items and how they interact with

various assistive technology software and devices Infrastructure Trials and Technical Assistance Bulletin

Student Needs and the Technical Landscape

The job we do in education is important. Students rely on us to be the gatekeepers of quality and high expectations. They rely on us to help them maintain their hope and achieve their dreams.

Our responsibility to students

Given the right tools …

People with disabilities …can acquire the same

information

…participate in the same activities

…be active producers as well as consumers

Building AssessmentsWhen we build assessments, we have to juggle a lot of competing priorities. It’s not an easy thing to do, but we have to learn from our experiences and make improvements in our process as we go. If we keep doing things the same way, then something bad is eventually going to happen....

Building Assessments

Building assessments requires us to work with diverse groups of people and it requires a lot of different types of expertise. This can be challenging and because of these challenges, it can be tempting to make assumptions about student needs. Pressure and severe time constraints are part of building assessments, so we need a plan to keep us on track and to ensure that we’re not making avoidable missteps.

Building AssessmentsHaving a plan is USUALLY better than juggling … but if we have a bad plan then we’re still not going to get where we need to go. We have to think differently about assessments now that we can leverage the power of technology. Building digital assessments in the same we’ve always built paper assessments is not an improvement. We have to take responsibility to understand our users and the technical landscape of possibilities.

Building Assessment

When a good plan comes together, it’s because it’s built on a solid foundation. The corner stone of a solid assessment foundation, is truly understanding the users.

Understanding the population

10% - 15%

According to statistics, the community of people with disabilities is the is the largest minority in the world. Currently, 10 – 15 percent of the world or over 600 million people live with a sensory, intellectual or physical disability.

Overview of Global Landscape

United States 9.9% aged 16-64 35% work compared

with 78% 2 times more likely drop

out of high school Approximately 13%

student population 62% spend at least 80%

of school day in general education classroom

Globally 98% don’t attend school 2/3 of people with

disabilities live in low income countries

85% under age 15 are from developing nations

2025 the number of people with disabilities will double

Specialized Tools

Some Examples Include: Refreshable Braille Displays, Screen Readers, Screen Magnifiers, Closed Captions, Sign Language Videos, Text-to-Speech with Synchronized Highlighting, Augmentative Communication Devices, and even something as simple as a pair of eye glasses

Simply put …

If you don’t know your audience: you can’t

effectively assess them and

you can’t effectively inform changes in instruction

Assessment Definitions Features for All / Universal Tools - access features of the

assessment that are available to all students on the basis of student preference and selection (e.g. highlighter, line reader, etc.)

Accessibility Features Identified in Advance / Dedicated Supports - features that a student regularly uses in the classroom for instruction and/or assessment for whom the need has been indicated by an educator or group of educators, including an IEP or Section 504 team. (text-to-speech for math, color contrast, etc.)

Accommodations - changes in procedures or materials that increase equitable access during the assessments. (Braille, Large Print, Assistive Technology, etc.)

UDLUniversal Design has 3 principles that work together to support access for student learning. The first principle is representation which is how the information is presented. The second is action and expression which gives students alternatives to pencil and paper to express the knowledge on a subject and the third is engagement which stimulates student interest by providing learning in a variety of ways. It is important to note that it might not be feasible to incorporate all three principles into every lesson every time. Rather, they are intended to guide instruction over time. Even when teachers apply the three principles, some students may need additional support. Consequently, teachers will sometimes have to make accommodations (e.g., allow the use of a spell checker) to meet an individual student's needs.The important thing about UDL is that is supposed to be done BEFORE designing the lesson. That is what is important for our discussion today. The current design review process is a start in the right direction but it will not be enough as we move forward. New accessibility requirements go beyond current UDR considerations.

It’s a principle It is NOT a

standard

Representation

EngagementAction and Expression

It is irresponsible to talk about Universal Design for Learning without emphasizing that you cannot have a universally designed product that does not adhere to accessibility standards.

With great powercomes great responsibility.

Standards Conundrum

APIPAccessible

Portable ItemProtocol

QTIQuestion and Test

Interoperability

WCAGWeb Content Accessibility

Guidelines

Accessible PortableItem Protocol (APIP)

APIP Specification – IMS Global XML based specification that allows computer-

based assessments to be portable and accessible

Accommodations are built into the test delivery system

Personal Needs and Preferences (PNP) profile

http://www.imsglobal.org/APIP/index.html

Question & Test Interoperability (QTI)QTI Specification – IMS Global

Provides standard XML language

QTI ensures interoperability between systems

QTI has been used internationally for over a decade.

Not QTI compliant = Not APIP compliant

http://www.imsglobal.org/question/

Web Content Accessibility Guidelines (WCAG)

Internationally recognized accessibility web standards

Guidelines allow content and applications to be

accessible to people who depend on the use of

assistive technologies

ISO Standard #: ISO / IEC 40500:2012

12 Guidelines with 61 measurable Success Criteria

In summary …

APIP = building the access features into the testing platform

QTI = ensuring that XML versions of test items can be stored in banks and shared with other QTI-compliant testing systems.

WCAG = ensuring that third-party assistive technology tools can work with the testing platform

Challenges APIP: ensuring that all of the accommodations

students need are present within the system, particularly when students have complex access needs

WCAG: ensuring that test security and construct validity are maintained when third-party access tools are introduced to the testing environment.

QTI: there is a limited set of HTML elements included in QTI

Web Accessibility Initiative – Accessible Rich Internet Applications (WAI-ARIA)

WAI-ARIA Specification (W3C) Defines a way to make Web content and Web

applications more accessible to people with disabilities.

It especially helps with dynamic content and advanced user interface controls developed with AJAX, HTML, JavaScript, and related technologies.

Our Accessibility Journey In May 2014, after going full

speed down the APIP road for the previous 12 months, we discovered that our accessibility “solution” – including our Testnav 8 Client technology and our content encoding model (derived exclusively from QTI/APIP) – was ineffective for people with certain disabilities.

Why Ineffective? Lack of Fidelity

Standard Text-to-Speech implementations do not offer sufficient Semantic and Perceptual “visibility.”

Lack of Authenticity Standard Text-to-Speech implementations (e.g., clicking links and

listening to time-coded audio files), are not “authentic.” Using Assistive Technology (AT) such as JAWS, NVDA, and others,

is the authentic method used by students every day. Denying access to these kinds of tools leads to further unfairness.

Assistive Technology (AT) functions and features are tightly coupled to HTML patterns that are dying or dead. Even if we wanted to support authentic AT, doing so using common

conventions with browser technology is not easily achievable with the design of modern, hi-performance web applications such as TestNav 8.

About APIP APIP

is not effective for providing content fidelity for people with certain disabilities who use certain AT

is useful and sufficient for describing certain kinds of accommodations

is comprised of two accessibility-related xml vocabularies:

1. Personal Needs Profile (PNP) <accessForAllUser>

2. Item/Test Accessibility Info <apipAccessibility>

Limitations of <apipAccessibility> Does not account for the way some users; i.e., visually

impaired users of assistive technology “see” content using the authentic tools that are commonly used; e.g., JAWS, NVDA, ZoomText, etc.

Ineffective for providing random access to content. In other words, no good way to encode “semantic markers.”

Ineffective for providing “perceptual visibility” within Item content and Item Interactions.

APIP PNP (Personal Needs & Preferences)

PNP – A LOT of “goodness.” Standardizes ways of describing Student-Assessment accommodations.

<accessForAllUser>

<control>

<apip:apipControl>

<apip:additionalTestingTime>

<apip:assignedSupport>true</apip:assignedSupport>

<apip:timeMultiplier>1.5</apip:timeMultiplier>

</apip:additionalTestingTime>

<apip:breaks>

<apip:assignedSupport>true</apip:assignedSupport>

</apip:breaks>

</apip:apipControl>

</control>

<display>

<screenEnhancement>

<magnification>3</magnification>

</screenEnhancement>

</display>

</accessForAllUser>

Accommodation Fidelity

APIP WCAG + ARIA

• So let’s talk for a minute about how students with disabilities overcome barriers to digital content in the context of assessment.

• On the screen you see a pair of crutches that for illustrative purposes are representing the APIP standard.

• You also see a photograph of a girl in a customized wheelchair. This photo is meant to illustrate WCAG + ARIA standards and best practices, which testing companies can comply with in order to allow students to use third-party assistive technology

• The crutches have been decorated with bows and decorative tape. It looks like they may have added some additional padding to the top of the crutches. You can also see that the crutches can be made taller or shorter, depending on the height of the user. In the same way that built-in accommodations through APIP are meant to help learners overcome minor barriers to accessing digital content, these crutches are designed to help people overcome the effects of a minor leg injury. Lots of people could use these crutches or similar crutches that perhaps are decorated in a different way. They could use the crutches with very little training and even though it might be slightly annoying to have to use crutches, you could at least still be mobile and accomplish a lot of tasks.

• However, a person who needs a power wheelchair, will never be able to use a pair of crutches. They need a much more robust set of supports in order to accomplish what other people might accomplish with the crutches or without any support at all.

Accommodation Authenticity• And … the person who needs a power wheelchair, needs that chair to be

customized to their unique needs. You will notice that the girl in this wheelchair, has a head rest, a bar across the middle of her chair to help her stabilize herself and she has a mechanism to hold her legs in place. In the same way, students who use complex assistive technologies will often customize how those tools work so that they can meet their unique needs. Students learn to use these complex tools to a level of muscle memory so that they don’t even have to think about how the tool works as they use it to complete their school work.

• If we try to build in a complex access system, like a power wheelchair, no matter what we do it won’t be the same tool that students use every day with their customized settings. It won’t be a tool that students can use without having to think about how to use it. Notice that the other power wheelchair on this screen does not have the head set or the mechanism to hold the users legs in place, nor does it have a bar across the middle to help with stabilization.

• This is meant to illustrate that if testing companies try to build in complicated access systems into an online assessment system, they may meet the needs of some students, but because they have taken away the students’ ability to choose their access system through third-party assistive technology, they have created a man-made barrier to success for students who’s access needs exceed or differ from what is built into the system.

• My point is that there is a limit to what we can do with APIP and we, as a company, are continuing to explore where APIP ends and WCAG and third-party Assistive Technology begin

The One-Page WebappImportant AT features are tied to obsolete HTML patterns

All existing WCAG / ARIA examples (that we found) use old-school html page / form loads to set the stage for AT behaviors.

Modern, hi-performance web applications, such as TestNav 8, load the entire application once, and abandon old-school html form, fieldset, and submit patterns.

Without forms and fieldsets, a lot of AT stops working. This requires a great deal of engineering to work around assumptions made by AT.

Standards Toolkit

APIPAccessible

Portable ItemProtocol

QTIQuestion and Test

Interoperability

WCAGWeb Content Accessibility

Guidelines

Plus …WAI-ARIA!!!

Example – Two-Column Item

A two-column assessment item. The left column contains text outlining the assessment problem for the student. The student is to read the math word problem and select the correct answer from the multiple choice answers given in the right column. The right column contains four multiple choice answers labeled A, B, C, and D. Each multiple choice answer is a graphic image of a wrapped box with colored wrapping paper. Each present has a different color paper.

The human eye naturally scans this from top to bottom, left to right, “chunking” the content.

Application/Content designers add/remove visual importance to screen elements so as to draw the eye to certain elements.

The human eye can - and does - jump back and forth to “randomly access” the left and right columns.

Users of AT can do none of this without the injection of semantic markers into content

HTML Content through the lens of AT<div class="row"><div class="span6"><p>Sarah has eight dollars. She wants to buy as many boxes of chocolates as she can.</p><p>The blue box of chocolates cost two dollars and the red box cost three dollars. The yellow box of chocolates is three dollars more than the blue box. The purple box costs twice as much as the blue box.</p><p>If Sarah spends all eight dollars and only buys two different boxes of chocolates, which chocolates can she buy? Select the two boxes of chocolates that Sarah buys.</p></div><div class="span6"><div class="well"><div class="whitebg pad8"><ul class="int-choice-list two-up at-enable" data-module="ChoiceInteraction“ data-identifier="RESPONSE"><li><label class="checkbox" for="choice_cdf06"><div class="int-choice-control"><input type="checkbox" id="choice_cdf06" class="checkbox" name="group_RESPONSE" value="B" /></div><div class="int-choice-label">A.</div><div class="int-choice-desc"><span><img src="public/assets/bluebox.png" alt="blue box" width="88" height="100" /></span></div></label></li><li><label class="checkbox" for="choice_3230c"><div class="int-choice-control"><input type="checkbox" id="choice_3230c" class="checkbox" name="group_RESPONSE" value="R" /></div><div class="int-choice-label">B.</div><div class="int-choice-desc"><span><img src="public/assets/redbox.png" alt="red box" width="72" height="100" /></span></div>

</label></li><li><label class="checkbox" for="choice_a8002"><div class="int-choice-control"><input type="checkbox" id="choice_a8002" class="checkbox" name="group_RESPONSE" value="Y" /></div><div class="int-choice-label">C.</div><div class="int-choice-desc"><span><img src="public/assets/yellowbox.png" alt="yellow box" width="102" height="100" /></span></div></label></li><li><label class="checkbox"

for="choice_c79b2"><div class="int-choice-control"><input type="checkbox" id="choice_c79b2" class="checkbox" name="group_RESPONSE" value="P" /></div><div class="int-choice-label">D.</div><div class="int-choice-desc"><span><img src="public/assets/purplebox.png" alt="purple box" width="105" height="100" /></span></div></label></li></ul></div></div></div></div></div>

Semantic Markers Semantic Markers are elements within the html

markup that content authors can add in order to describe structure and meaning to assistive technology.

Semantic Markers are akin to the Table of Contents of a book. They provide an outline view of content.

WCAG / ARIA to the rescue! Three common strategies for adding semantic markers: Use header tags Use anchor tags Use ARIA “landmark” roles

What Does the QTI Look Like?<div class="row"> <div class="span6"> <h2 class="semantic-marker visuallyhidden">Introduction</h2> <p>Sarah has eight dollars. She wants to buy as many boxes of chocolates as she can. </p> <p>The blue box of chocolates cost two dollars and the red box cost three dollars. The yellow box of chocolates is three dollars more than the blue box. The purple box costs twice as much as the blue box.</p> <h2 class="semantic-marker visuallyhidden">Question</h2> <p>If Sarah spends all eight dollars and only buys two different boxes of chocolates, which chocolates can she buy? Select the two boxes of chocolates that Sarah buys </p> </div> <div class="span6"> <div class="well"> <div class="whitebg pad8"> <h2 class="semantic-marker visuallyhidden">Interaction</h2> <choiceInteraction class="two-up" maxChoices="0" responseIdentifier="RESPONSE" shuffle=“false"> <simpleChoice identifier="B"><span><img alt="blue box" height="100" src="assets/bluebox.png" width="88"/></span> </simpleChoice> <simpleChoice identifier="R"><span><img alt="red box" height="100" src="assets/redbox.png" width="72"/></span> </simpleChoice> <simpleChoice identifier="Y"><span><img alt="yellow box" height="100" src="assets/yellowbox.png" width="102"/></span> </simpleChoice> <simpleChoice identifier="P"><span><img alt="purple box" height="100" src="assets/purplebox.png" width="105"/></span> </simpleChoice> </choiceInteraction> </div> </div> </div></div>

Random Access

Three heading H2 tags of “<h2>Introduction</h2>”, “<h2>Question</h2>”, and “<h2>Interactions</h2>” that feed into assistive technology and then to a computer.

Semantic Markers provide the outline view of the Content. All modern AT “sees” heading tags and anchor tags. Users can

access these with “quick key” shortcuts, enabling AT users to discover the structure and meaning of the Content.

This gets us closer to random access of content, thus providing a more equitable experience.

Perceptual Visibility Semantic Markers are usually not enough. We need to add “Perceptual Visibility” to the

content. How? With ARIA.

Example – Describe the Interaction:

1. The interaction type; e.g., multiplechoice, matching, text entry, hotspot, graphing, etc.

2. The number of options

3. Option descriptions

4. Interaction context

Note: this can get very complex and sophisticated!

Four multiple choice answers labeled A, B, C, and D. Each multiple choice answer has an image of a box with colored wrapping paper. Each box is covered in a different color of paper.

<div class="row"> <div class="span6"> <h2 class="semantic-marker visuallyhidden">Introduction</h2> <p>Sarah has eight dollars. She wants to buy as many boxes of chocolates as she can. </p> <p>The blue box of chocolates cost two dollars and the red box cost three dollars. The yellow box of chocolates is three dollars more than the blue box. The purple box costs twice as much as the blue box.</p> <h2 class="semantic-marker visuallyhidden">Question</h2> <p>If Sarah spends all eight dollars and only buys two different boxes of chocolates, which chocolates can she buy? <span aria-hidden=“true”> Select the two boxes of chocolates that Sarah buys </span></p> </div> <div class="span6"> <div class="well"> <div class="whitebg pad8"> <h2 class="semantic-marker visuallyhidden">Interaction</h2> <choiceInteraction class="two-up" maxChoices="0" responseIdentifier="RESPONSE" shuffle=“false“ aria-label=“Select the two boxes of chocolates that Sarah buys. Multiple choice checkbox group with 4 options” role=“group”> <simpleChoice identifier="B“ aria-setsize="4" aria-posinset="1"><span><img alt="blue box" height="100“ src="assets/bluebox.png" width="88"/></span></simpleChoice> <simpleChoice identifier="R” aria-setsize="4" aria-posinset=“2"><span><img alt="red box" height="100" src="assets/redbox.png" width="72"/></span></simpleChoice> <simpleChoice identifier="Y“ aria-setsize="4" aria-posinset=“3"><span><img alt="yellow box" height="100" src="assets/yellowbox.png" width="102"/></span></simpleChoice> <simpleChoice identifier="P“ aria-setsize="4" aria-posinset=“4"><span><img alt="purple box" height="100" src="assets/purplebox.png" width="105"/></span></simpleChoice> </choiceInteraction> </div> </div> </div></div>

What Might the QTI Look Like?

Semantic and Perceptual VisibilitySemantic Markers and ARIA, when used together, get us closer to the best of both worlds.

1. Provide big picture meaning of content.

2. Random access to chunks of content, saving time, reducing cognitive load.

3. Fine-grained, perceptual detail, where needed.

…all of the above using the authentic AT that students use every day in classrooms and at home. APIP, alone, accounts for very little of this.

Progressive Enhancement

Drag and Drop

This emulates Matching with paper.

On the screen you see a picture of a sample drag and drop item. There are four “draggers” labeled “Capulet, Demetrius, Lysander, and Prospero.” There are three drop bays, labeled, “A Midsummer-Night’s Dream, Romeo and Juliet, and The Tempest.” The purpose of the item is to match the character with the play in which they appear.

The same information can be presented to screen readers as a table with 4 columns and 5 rows. In this example, the names of the plays are listed in the first row as: A Mid-Summer Night’s Dream; Romeo and Juliet; The Tempest

The names of the characters are listed in the first column as: Capulet; Demetrius; Lysander; Prospero

The data in the top row and the first column have header tags to indicate that this data would be presented to students through a properly marked up HTML table. The data cells in this example have radio buttons, but depending on the construct, they could be checkboxes instead.

In an actual HTML table, the data cells would also be marked up, so this is just for illustrative purposes. With a properly marked-up HTML table students can use their screen readers and/or refreshable Braille devices to have randomized access to the table content.

Data Structure + Look and Feel + Functionality

Sample Table

<th scope=“col”>A Mid-Summer

Night’s Dream</th>

<th scope=“col”>Romeo and Juliet</th>

<th scope=“col”>The

Tempest</th>

<th scope=“row”>

Capulet</th>

<th scope=“row”>

Demetrius</th>

<th scope=“row”>

Lysander</th>

<th scope=“row”>

Prospero</th>

EVERY Student Gets the Whole EnchiladaOn the screen you see a plate filled with enchiladas. Each enchilada is unique. This is to illustrate that … If the diverse needs of students are considered

at the beginning of the design, item content can be built without compromising: Accessibility Rigor Interactivity Construct validity Security

What Happens with Progressive Enhancement? In terms of our drag and drop example:

Fully sighted participants with good motor ability can drag answers to the proper drop bays.

Other participants with certain types of assistive technology can use the table directly.

Everybody wins. AND, if we need a new interface we can build

it from the table.

Interdependencies

WCAG

ATAG UAAG

W3C Guidelines Content Authoring User Agents

Call To Action

Unfortunately…

1. For taxpayers,

2. For users of assistive technology

3. Providers of assistive technology, and

4. For assessment providers,

…it’s just not that easy!

Why? Because the ARIA integration of Browser technology with the Accessibility API and with Assistive Technology is very uneven and often unpredictable.

Unique Opportunities for Computer-based AssessmentsDave Edyburn, Professor

University of Wisconsin-Milwaukee

Ruben R. Puentedura, Ph.D explains that the goal of technology integration is to have students create content and learn in a technology rich environment with opportunities to apply their critical thinking skills. He outlines four levels of technology integration in the SAMR Model. SAMR stands for Substitution, Augmentation, Modification, and Redefinition. As technology integration progresses, the effects on student learning outcomes move from enhancement to transformation

Ehancement: Substitution – Technology acts as a direct tool substitute, with no

functional change Augmentation – Technology acts as a direct tool substitute with

functional improvement Transformation:

Modification – Technology allows for significant task redesign Redefinition – Technology allows for the creation of new tasks,

previously inconceivable

SAMR Model

Missed Opportunities

On the screen you see a name tag that says, “Hello, my name is Missed Opportunities.” We have missed:

The opportunity to develop a system that generates empirical data about the effects of various interventions on student performance. This would overcome the historical problems identified in the research literature that the lack of validity of teachers selecting accommodations for students.

The opportunity to design a system that would obtain value added data about student performance.

The opportunity to expand access. Not just physical and sensory access but cognitive access.

Workspace …

MY Roles

Responsibilities

Interests

Education

Abilities

Expertise

Habits

Used Items

MY Team

Manager

Colleagues

Competition

Partner

MY Process

Projects

Tasks

Activities

Decisions

Tools

Theory of the User

Cartoon with two dogs. The first dog is sitting in a desk chair in front of the computer, looking down at the second dog who is sitting on the floor. The first dog says, “On the internet, no one knows you’re a dog.”

The term theory of user refers to system analytics that are used to analyze and profile the user.

We are all familiar with the recommendation engines at Amazon and Pandora that make recommendations for us based on our historical preferences.

Why can't these interventions be applied to digital assessment? For example, can we determine/verify whether or not the user has a disability? That is, what type of interaction and response patterns are indicative of a

disability? Or, is the student an English Language Learner?

Embedded Support

• Given the purpose of assessing whether students are college and career ready, it is inappropriate to create assessment environments where students do not have access to the tools they routinely use.

• One way of thinking about how to implement universal design in the assessment context is known as embedded support.

• That is, tools such as a notepad, calculator, dictionary, etc. are embedded in the system. Embedded supports can be activated one of three ways

• always on for everyone• available only to students who prequalify, and• activated based on student performance

• The availability and use of embedded supports requires a new generation of psychometrics.

A New Generation of Psychometrics

The transition from paper-based assessments to digital assessments requires that we consider new approaches to establishing reliability and validity. Indeed, the field of psychometrics is likely to be transformed though learning analytics.

For example, dynamic norming is a term used to describe the process of creating norm groups based on demographics of low incidence groups (i.e., disability, use of specialized supports like assistive technology) to make performance comparisons that would never be possible with traditional instrument validation because of the extreme cost associated with identifying and reaching such low incidence populations.

Discussion Points

Measuring What Students Know and Can Do Purpose – gather information related to target

construct, while minimizing the construct-irrelevant variance, to make valid inferences

Techniques Universal Design for Learning Accessibility and Accommodations Framework

Always available Identified in advance for student Accommodations Embedded or non-embedded

Item modification (Progressive Enhancement) Accessibility Reviews

Understanding Risks … The legal landscape is changing

Section 508 of the Rehab Act is being revised. The foundation of the revision is the Web Content Accessibility Guidelines (WCAG), version 2.0, Level AA

The Americans with Disabilities Act (ADA ) is being revised. Title III of the ADA

Assessment: Places of “Public Accommodation”

Department of Justice has made it clear that they are interpreting web-based services as places of public accommodation (see recent edX settlement – inaccessible LMS)

OCR has defined “effective communication” as: Timeliness of delivery Accuracy of the translation, and Provision in a manner and medium appropriate to the significance of the

message and the abilities of the individual with the disability

Advocacy Groups:Entering Into the World of K-12 Education Through the Assessment Door… - For the past several years, advocacy groups, such as the National

Federation of the Blind (NFB) have focused the majority of their litigation efforts in higher education.

- They are now entering into the world of K-12 education through the assessment door.

- In October 2014, the NFB posted a notice to their listserv, asking families to contact them if they or their children have experienced inadequate practice opportunities for any high stakes assessment.

In November of 2014, the Texas Chapter of the NFB passed a resolution to make sure that the Texas state assessment is accessible.

- The NFB’s legal team is presenting to parents at the National Convention on July 5th about what they need to know to properly advocate for their children in instruction and assessment.

- They have learned a great deal about assessment over the past year and a half and they know where the weaknesses are in the industry.

What can states do? Do your research:

Know the needs of students with various disabilities

Know the tools they use to access digital contentKnow the laws that protect them

Make sure your RFP language accurately spells out accessibility requirements and that you specify quantitative measures for accessibility (e.g. WCAG 2.0, Level AA conformance)

Moving forward … Challenges

Standardization – No longer one-size-fits all Consistency between classroom and testing environments Changing technology 3rd party technologies not following W3C standards APIP-WCAG-QTI toolkit

Future Continued move to online assessment (Pearson –25% in 2012 to over

50% projected for 2016) Research on effects of use of accessibility features Data analytics and psychometric models

Accessibility standards matters They are our front-line defense for students with disabilities to ensure

that they have equality in education and assessment.

Questions