University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessible Education through Crowd-Powered Systems

Jeffrey P. Bigham @jeffbigham

University of Rochester

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessible Education through Crowd-Powered Systems

Jeffrey P. Bigham @jeffbigham

University of Rochester

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Dual Goals

1. Advocate for Accessibility

2. Accessibility w/ Crowd-Powered Systems

Accessibility

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

(lower bounds on) Disability in Education

• 15% of the world’s population is disabled (about 1 billion people)

• 9% of the school-age population have disabilities

• 13% of undergraduate IT majors have disabilities. – 5% of graduate IT majors have disabilities.

• 0.8% of IT doctorates have disabilities (53 between 1999 and 2004)

• 40% of students with disabilities complete a bachelor's degree

– Compared with 60% of everyone else

Accessibility

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe Accessibility

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

$150 / hour for real-time captioning

hours to convert book to digital text - days to convert its figures

adapting assignments

Accommodation is Expensive and Slow

Accessibility

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Crowd Education

• Prepared Resources

– fix a problem once, reach many students

• More (distributed) Interaction

– as students support students, less accommodation?

• Student Crowds

– students have the expertise to help

Accessibility

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe Accessibility

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

VizWiz Nearly Real-Time Answers to Visual Questions Real-Time

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

VizWiz

Bigham et al. Nearly Real-Time Answers to Visual Questions. UIST 2010.

VizWiz

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Releasing VizWiz

• Released on May 31, 2011

– 5000 users asked more than 50,000 questions

– average time to first answer < a minute

VizWiz

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe VizWiz

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Legion:Scribe Real-Time Captions by Groups of Non-Experts

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Real-Time Captioning

Stenographers ASR

expensive cheap

difficult to schedule available on demand

lack domain expertise can be trained for new vocab

pretty accurate does not work

Can I help?

NO, you are worse than ASR.

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Real-Time Captioning

Lasecki et al. “Real-Time Captioning by Groups of Non-Experts.” UIST 2012.

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Scribe Algorithm

Worker 1

Worker 2

Worker 3

open the

open java

the java

file up and

Graph

Time

Stage 1 Stage 2 Stage 3

file now

fiel

Baseline open the java file now and

openjava

the

openjava

the

openjava

the

filenow

up

and

W.S. Lasecki, C.D. Miller, D. Borrello and J.P. Bigham. “Online Sequence Alignment for Real-Time Audio Transcription by Non-Experts.” AAAI 2012 (poster).

Multiple Sequence Alignment

Online Version

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Scribe Interface

Encourages: - real-time input - global coverage - short sequences

Co-evolution of Interface and Algorithm

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Coverage Graph

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Tradeoff

Failures: “n-factorial” “in pectoral”

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Interesting Qualities

• Captionists can be experts – not at captioning but in the subject

• Low cost – $30/hour on Mturk (did not optimize)

– or free (impossible before)

• Recruited on demand – for only as long as needed

Scribe

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Summary

• To be open to all, content must be accessible

• Scale Helps – Produced once, can be reused by many (usually)

– Many students ~= many experts

• Making interactive content accessible – leveraging the student crowd

ç

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe

Thanks!

hci.cs.rochester.edu @jeffbigham

Funded by: National Science Foundation Grants (#IIS-1149709, #IIS-1116051, #IIS-1049080 ), and Google.

ç

University of Rochester Human-Computer Interaction Jeffrey P. Bigham

Accessibility VizWiz Scribe