Christopher BaileyDr. Elaine Pearson

Teesside University

c.p.bailey@tees.ac.uk

Evaluation of the Effectiveness of a Tool to Support Novice Auditors

Practitioner Context

• Computing students need support with accessibility as:

– Lack general awareness of accessibility. – Minimal inclusion in UG and/or PG curriculum.– Evaluation is only one element of accessibility.– Projects include developing live websites. – Limited face-to-face student/tutor instruction.– Limited time to dedicate to accessibility.– Limited access to expertise/end users for testing. – Need skills to enhance employability.

Research Context

• Evaluation report (audit) has motivational and educational value (Sloan, 2006).

• Manual evaluation is important (WCAG 2.0, UWEM, BW).

• Evaluation support tools aimed at experienced evaluators.

• The expertise level of the evaluator is particularly significant (WCAG 2.0, Brajnik, 2010; BW, Yesilada et al, 2009).

• Fewer false positives and false negatives, faster, and more confident in judgements.

• Comprehension, Knowledge and Effort (Alonso et al, 2010).

Accessibility Evaluation Assistant• Educational evaluation support tool for novices

• Support functions:– User Group– Site Features– Check Categories

• Structured Walkthrough Method – Translation of expert process– Title and Summary of Accessibility Principle (Heuristic)– The User Group(s) affected– The nature of problem caused and barrier presented– A step-by-step checking procedure (manual and/or tool support– Guidance for verification (interpreting results of tool)– An example video tutorial

How Effective is AEA?• Initial trial showed promising validity and reliability (Bailey &

Pearson, 2011)

• Compare relative effectiveness to WCAG 2.0

• Reliability (reach same decision)

• Validity– Correctness (in matching experienced evaluator)– Sensitivity (identifying true barriers)

• Usefulness

• Usability

• Efficiency

Trial Methodology• 38 Undergraduate Computing Students, 12 week elective

Accessibility and Adaptive Technology Module.

• Conducted within constraints of curriculum as an assessment

• 4 Tasks:– 2 Evaluation Exercises: Sunsail and Harley Davidson Home

Pages– 2 Reflective Pieces: Personas/User Group, Experience of

Evaluation (Compare Methods, Describe Problems).

• Evaluate 15 AEA Heuristics, equivalent 17 WCAG 2.0 Success Criteria, relevant to both webpages.

• Check criteria is Met, Not Met or Partly Met and explain/justify their decision.

Results: Reliability

Grp.Week 1 Week 2

Method/Website Rel. Method/Website Rel.

1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 63%

2 WCAG 2.0 – Harley Davidson

67% AEA - Sunsail 78%

3AEA - Sunsail 71%

WCAG 2.0 – Harley Davidson

59%

4 WCAG 2.0 - Sunsail 70% AEA - Harley Davidson 63%

Method

Week 1 Week 2 Overall

AEAWCAG

2.0AEA

WCAG 2.0

AEAWCAG

2.0

Reliability 72% 68.5% 70.5% 61% 71.25% 64.75%

Results: Validity (Correctness)

Grp.Week 1 Week 2

Method/Website Validity Method/Website Validity

1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 49%

2 WCAG 2.0 – Harley Davidson

59% AEA - Sunsail 73%

3 AEA - Sunsail 66% WCAG 2.0 – Harley Davidson

49%

4 WCAG 2.0 - Sunsail 50% AEA - Harley Davidson 62%

Method

Week 1 Week 2 Overall

AEAWCAG

2.0AEA

WCAG 2.0

AEAWCAG

2.0

Validity 69.5% 54.5% 67.5% 49% 68.5% 51.75%

Results: Validity (Sensitivity - AEA)

Sunsail Home Page

Heuristic Failure Detection

Images of Text 72%

Text Size 89%

Keyboard Navigation 94%

Skip Navigation 89%

Image Text Alternatives

94%

Headings 83%

Form Labels 39%

Valid (X)HTML Code 94%

Site Map 100%

Average 85%

Harley Davidson Home Page

Heuristic Failure Detection

Images of Text 88%

Colour Contrast 59%

Text Size 88%

Keyboard Navigation 76%

Skip Navigation 94%

Image Text Alternatives 82%

Headings 76%

Form Labels 76%

Identify Language of Page

76%

Valid (X)HTML Code 100%

Search Function 94%

Average 83%

Results: Validity (Sensitivity)

• Overall Comparison

Sunsail Home Page

Method Average Detection Rate

AEA 85%

WCAG 2.0 80%

Harley Davidson Home Page

Method Average Detection Rate

AEA 83%

WCAG 2.0 74%

Qualitative Feeback (AEA) • Usable, Useful, Efficient?

Positive Negative

Easy/Simple to Use/Concise Check explanation too brief

Easy to Understand/Clear Terminology

Poor UI Usability/Videos too small

Explanation Guides User Checks Require Individual Judgement

Categorisation/Grouping of Checks

Hard to Judge Met/Not Met

Speed of Check Process No Advice on Solving Problems

Helpful Videos Bugs and Errors

Qualitative Feeback (WCAG)

Positive Negative

Detailed Explanations Confusing/Difficult to Understand

Linked to Regulation and Industry

Complex/Hard to Use

Real Examples in Documentation

More Knowledge/Experience Required

Fewer Individual Judgements Required

Hard to Judge Relevance of Check

Easy to Navigate Documentation

No Explanation for Performing Check

• Usable, Useful, Efficient?

Findings and Implications• Use of AEA leads to higher levels of reliability and correctness, and

novices are able to identify true barriers.

• Subjective judgements influence results (evaluator effect).

• We can provide a method, but we still can’t control:– Thoroughness– Effort– Competence

• Novices can relate heuristics to WCAG 2.0 Success Criteria

• Structured approach makes it easier to apply, repeat and learn.

• Complement existing methods for use in industry.

Future Work• Enhance Structured Walkthrough approach to further limit errors:

– Remove ambiguities to reduce misinterpretation– Enhance guidance for decision making– Develop a simple metric for determining severity

• Redevelop AEA to increase flexibility of delivery

• Trial in another institution

• Add functionality to formally support WCAG 2.0 evaluation

• Longitudinal studies:– Further demonstrate effectiveness of AEA– Barriers which novices consistently identify/miss– Identify best strategies to teach novices (evaluation, simulation,

demonstration, personas)

Student Experience

• “….using the AEA are that you are told precisely what you are checking clearly and concisely, making the process faster and simpler. With WCAG 2.0 the sentences are extremely long…. I had to repeatedly read the check.”

• “The AEA is very easy to understand, follow and implement the checks. The step-by-step instructions are not only helpful, but informative and made my understand why I should be performing the check”.

Student Experience

• “The advantages of the AEA tool are speed, simplicity and ease-of-use. Using the AEA definitely felt more effective as it follows the WCAG 2.0 guideline but streamlines the majority of checks allowing you bypass the technical jargon and bureaucracy in WCAG.”

• “I found the AEA tool significantly easier and more intuitive that the WCAG 2.0 approach….it guides the tester to a relevant section, explains what to look for in an easy to understand manner as well as explaining the purpose of the test.”

Christopher BaileyDr. Elaine Pearson

Teesside University

c.p.bailey@tees.ac.uk

Evaluation of the Effectiveness of a Tool to Support Novice Auditors

http://arc.tees.ac.uk/aea