Assessing the Effort of Repairing the Accessibility of Web Sites

ASSESSING THE EFFORT OF REPAIRING THE ACCESSIBILITY OF WEB SITES

Nádia Fernandes, Luís CarriçoUniversity of Lisbon

ICCHP, Linz, Austria, July 11-13, 2012

ICCHP, Linz, Austria, July 13, 2012 2


Introduction• 40-50% of the Web content uses templates.

• Evaluations are performed in pages as a whole.

• Obfuscating results.

• Available metrics may be misleading.


Objectives1. Assess the effort of repairing a site’s accessibility that was

originally developed using templates

Requirements:

• Develop a new metric, • Template detection algorithm

2. Introducing accessible templates as a form of rapidly repairing a page


The Accessibility Repairing Effort Metric (AREM)

“Primitive elements”- elements of a page, when an element that is part of a template is only considered once.

The metric considers the sum of the number of fails and warnings reported by accessibility evaluation techniques, excluding repeated instances.


The Accessibility Repairing Effort Metric (AREM)

The purpose of this metric is:

1. assess the quality measurement of the accessibility of site construction (effort person.month);

2. and not the perceived quality of the site towards end-users.


Basis:

• QualWeb evaluator

• Fast Match algorithm

The Platform for Accessibility Evaluation


Procedure1. The DOM trees are obtained

2. The pages are compared using the Fast Match algorithm • Comparative function • Result: The “primitive” nodes

3. The accessibility evaluation is executed (QualWeb evaluator)

4. Reports generation (new reports)

5. The repairing effort estimation according to AREM is computed.

Experimental Study• Objective: understand the advantage of using AREM and

validate the template detection method.

• Object of study: 15 sites with templates (Alexa Top 100)


5 low level

5 medium level

5 high level


Experimental Study• QualWeb was applied with the template aware option set.

• The values for the AREM using the element primitiveness.

• We used a conservative rate metric to obtain values of accessibility quality to compare with AREM.

rate conservative =


Metrics Results – AREM (I)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

100002000030000400005000060000700008000090000

100000

Non-primitive PrimitiveWeb sites

1 – Higher level of template usage, 15 – Lower level of template usage


Metrics Results - Conservative rate (I)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150.000.050.100.150.200.250.300.350.40

Non-primitive Primitive

Web sites



Metrics Results – AREM (II)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

100002000030000400005000060000700008000090000

100000

Non-primitive PrimitiveWeb sites



Metrics Results - Conservative rate (II)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150.000.050.100.150.200.250.300.350.40


Web sites



Metrics Results - Conservative rate (III)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150.000.050.100.150.200.250.300.350.40


Web sites



Validating the Approach

Web sites % template % errors on template detection

Higher level of template detection

56% 14%51% 8%

Mid level of template detection

33% 6%31% 12%

Lower level of template detection

18% 7%15% 1%

The number of incorrect elements detected by Fast Match algorithm is less than 10% (average).


Discussion• The validation of the template detection yielded a deviation.

• The algorithm had more failures in sites which use more template based components.

• Regarding the AREM metric:• The difference between the computed values is high; • Less 30% of repairing issues (primitive elements);• Depending on the sites this value can decrease substantially.

• These results confirm and support our previous experiment’s results.


Conclusion• Templates can be very important, reducing the effort of

correction.

• The metric defined is a real indicator of the work that have to be done, unlike certain quality metric that can be misleading.

• We performed a validation experiment of both metric and framework and conclude that the template detection algorithm has a high efficacy.


Future Work1. Improvements of Fast-Match algorithm to guarantee a

higher accuracy level;

2. A large-scale evaluation of the fast match algorithm.


Thank you

[email protected]

Assessing the Effort of Repairing the Accessibility of Web Sites

Technology

Transcript of Assessing the Effort of Repairing the Accessibility of Web Sites