Strategies LLCTaxonomy

October 7, 2009 Copyright 2009 Taxonomy Strategies LLC. All rights reserved.

How to Validate a Taxonomy

ASIST Workshop Preview

2Taxonomy Strategies LLC The business of organized information

What is a Taxonomy?

v A categorization framework agreed upon by business and content owners (with the help of subject matter experts) that will be used to tag content.

6 broad, discrete divisions (called facets) 2-3 levels deep. Up to 15 terms at each level. 1200 terms total. With some logic—hierarchical, equivalent and associative relationships

between terms.

3Taxonomy Strategies LLC The business of organized information

Why taxonomy is important

4Taxonomy Strategies LLC The business of organized information

Why is taxonomy important?

v Easier information management.v Flexibility to respond to changing needs.v Foundation for findability and usability.

5Taxonomy Strategies LLC The business of organized information

How can you know if a taxonomy will work?

v Tagging content.v Publishing content.v Finding and using content in user-facing applications.

6Taxonomy Strategies LLC The business of organized information

What types of taxonomy validation can you do?

v Validating taxonomy design.v Validating taxonomy usability.v Validating taxonomy use in a collection.

7Taxonomy Strategies LLC The business of organized information

Taxonomy design validation methodsMethod Process Who Requires Validation

Web tool Open Card sort

Representative users

Rough taxonomy Card sort analysis (emergent patterns)

Cards & markup

Delphi card sort

Representative users

Rough taxonomy End result (consensus)

Walk-thru Show & explain

Stakeholders Rough taxonomy Approach Appropriateness to task

Walk-thru Check conformance to editorial rules

Taxonomist Draft taxonomy Editorial rules

Consistent look & feel

Matrix or cards

Closed card sort

End users (or surrogates)

Rough taxonomy Top queries, etc.

Consistent results

User satisfaction

Survey End users (or surrogates)

Draft taxonomy Reaction to taxonomy

Tagging samples

Tag sample content with taxonomy

Taxonomist Indexers

Sample content Draft taxonomy

(or better)

Content “fit” Fills out content inventory Training materials for people &

algorithms Basis for quantitative methods

8Taxonomy Strategies LLC The business of organized information

Usability validation methods

Method Process Who Requires ValidationUsability Testing

Task-based scenario testing (Find it, Tag it, etc.)

End users (or surrogates)

Draft taxonomy Tasks & answers

Tasks are completed successfully Time to complete task is reduced

User Satisfaction

Survey End users (or surrogates)

Draft taxonomy UI mockup,

Search prototype

Reaction to taxonomy Reaction to new interface,

Reaction to search results

9Taxonomy Strategies LLC The business of organized information

Collection analysis validation methods

Method Process Who Requires ValidationDistribution Statistical

analysis Taxonomist Analyst

Tagged collection

Do categories follow Zipf distribution?

Do % of categories equal % of documents?

Identify candidates for merging & splitting.

Query log & click trail examination

Clustering & statistical analysis

Taxonomist Analyst

Search logs Web analytics

Do (clustered) queries follow Zipf distribution?

Are top (clustered) queries in taxonomy?

10Taxonomy Strategies LLC The business of organized information

Full-day taxonomy validation workshop agenda

9:00-9:30 30 min Introduction9:30-10:00 30 min Taxonomy & validation overview

10:00-10:30 30 min Taxonomy design validation10:30-10:45 15 min Coffee Break10:45-11:30 45 min Delphi card sorting exercise11:30-12:00 30 min Closed card sorting

12:00-1:00 60 min Lunch1:00-1:45 45 min Closed card sorting exercise1:45-2:15 30 min Online tools2:15-2:45 30 Min Task-based validation methods2:45-3:00 15 min Coffee break3:00-3:45 45 min Task-based validation exercise3:45-4:15 30 min Collection analysis overview4:15-5:00 45 min Q&A, Closing

Hands-on Exercises BreaksLectures

11Taxonomy Strategies LLC The business of organized information

Closed card sorting case study

v Project: Develop Substance Abuse and Mental Health Services Administration (SAMHSA) taxonomy to be the backbone of a unified agency information service that collects, integrates, and reports on customer interaction data in ways that help the agency plan health communications products, messages and outreach.

v Task: Sort popular queries (words and phrases) from search logs into the most likely Taxonomy facet.

12Taxonomy Strategies LLC The business of organized information

Term sorting data collection form

Term Content

Type AudiencePopulation

Groups SubstancesConditions &

Disorders

Intervention & Treatment

TopicsProfessional &

Research TopicsGeographic Locations

Alcohol Anger Anxiety Autism Binge Eating Bipolar Disorder Change Management Child Co-occurring Disorders Counseling Depression Fulton County Health Literacy Immigrant Adolescents Inhalants Low Income Methadone Native American Culture Ohio Opium Posters PTSD Residential Treatment Rohypnol Smoking Stress Substance Abuse Suicide University Students Vicodin Webcast

13Taxonomy Strategies LLC The business of organized information

Summary of term sorting results

Term Content

Type AudiencePopulation

Groups SubstancesConditions &

DisordersIntervention &

Treatment TopicsProfessional &

Research TopicsGeographic Locations None

Alcohol 1 10 1 Anger 9 1 2 Anxiety 12 Autism 12 Binge Eating 10 1 1 Bipolar Disorder 12 Change Management 2 9 1Child 12 Co-occurring Disorders 11 1 Counseling 9 3 Depression 12 Fulton County 12 Health Literacy 12 Immigrant Adolescents 12 Inhalants 10 1 1Low Income 1 9 2 Methadone 10 2 Native American Culture 1 7 4 Ohio 12 Opium 12 Posters 9 1 1 1PTSD 12 Residential Treatment 1 11 Rohypnol 10 1 1Smoking 1 5 3 1 2 Stress 10 1 1 Substance Abuse 2 1 1 4 1 3 Suicide 1 7 1 3 University Students 12 Vicodin 11 1 Webcast 9 1 2

Frequently chosen related category Frequently chosen incorrect categoryCorrect category

14Taxonomy Strategies LLC The business of organized information

Percentage of popular search terms sorted correctly

Anxiet

yAuti

sm

Bipola

r Diso

rder

Child

Depre

ssion

Fulto

n Cou

nty

Health

Lite

racy

Imm

igran

t Ado

lesce

nts

OhioOpiu

mPT

SD

Univer

sity S

tude

nts

Co-o

ccur

ring D

isord

ers

Resid

ential

Trea

tmen

tVico

dinIn

halan

tsRo

hypn

olAlco

hol

Binge

Eating

Stre

ss

Chan

ge M

anag

emen

tPo

sters

Anger

Coun

selin

gLo

w Inco

me

Web

cast

Native

Am

erica

n Cult

ure

Suici

deSm

oking

Subs

tanc

e Abu

seM

etha

done

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

15Taxonomy Strategies LLC The business of organized information

Blind sorting of popular search terms (n=12)

50-60%(7%)

25-50%(6%)

< 25%(3%)

84% of terms were correctly sorted 60-100% of the time.

Results: Excellent

Difficultiesv For Methadone, confusion when, in this case, a substance is a treatment.v For general terms such as Smoking, Substance Abuse and Suicide, confusion about

whether these are Conditions or Research topics.

16Taxonomy Strategies LLC The business of organized information

Search terms sorting task user rating (n=12)

Easy42%

Easy/Medium25%

Medium33%

No one rated the task difficult!

17Taxonomy Strategies LLC The business of organized information

Task-based case study

v Project: Develop American Society of Civil Engineers (ASCE) taxonomy to draw from existing databases and content sources, and provision the next generation websites being built to support this non-profit, membership association’s activities.

v Task: Find specific content (web pages)

18Taxonomy Strategies LLC The business of organized information

Find web pages

A Audiences

C Content Types

E Event Types

L Locations

O Organizations

T Topics

T TopicsT.1 Architectural EngineeringT.2 Coasts & waterwaysT.3 ConstructionT.4 Cross-Cutting TopicsT.5 Disaster & Hazard

ManagementT.6 Education & Career

DevelopmentT.7 Engineering MechanicsT.8 EnergyT.9 EnvironmentT.10 Geotechnical EngineeringT.11 People, Projects & HeritageT.12 Planning & DevelopmentT.13 Professional IssuesT.14 Project ManagementT.15 Structural EngineeringT.16 TransportationT.17 Water & Wastewater

T.6 Education & Career Development

T.6.1 Continuing EducationT.6.2 Engineering EducationT.6.3 Management &

Professional DevelopmentT.6.4 Scholarships, Internships

& Competitions

ASCE Continuing Education http://www.asce.org/conted/

19Taxonomy Strategies LLC The business of organized information

Summary of navigation results trial

JF KSS MJ KS IM VM KSM GSB KCH

Trial 1 2 3 4 5 6 7 8 9 Key

Navigation

ASCE Continuing Education A > C > T A > O > C C > A > T C > E

Audience A.27

Content Type C.12

Event Type E.4 E.4 E.4 E.4

Location

Organization

Topic T.6.1 T.6.1 T.6.1

Difficulty Difficult Easy Difficult Easy Easy Difficult Medium Easy Medium Difficulty

Frequently chosen related category Frequently chosen incorrect categoryCorrect category

Gave up

20Taxonomy Strategies LLC The business of organized information

Overall navigation task performance (n=54)

Correct41%

Alternative46%

In-correct9%

Gave Up4%

v 87% navigated as predicted or used a reasonable alternative.v In only 4% of the trials, did the subject give up.

21Taxonomy Strategies LLC The business of organized information

Overall user rating of navigation task (n=9)

No one rated the overall task Difficult!

Easy33%

Medium67%

Strategies LLCTaxonomy

October 7, 2009 Copyright 2009 Taxonomy Strategies LLC. All rights reserved.

QuestionsASIST Annual Meeting - Full-Day Workshop

November 7, Vancouver, BC http://www.asist.org/Conferences/AM09/taxonomy.html

Gilbane Conference - Half-Day WorkshopDecember 1, Boston, MA

http://gilbaneboston.com/workshops.html#workshop

Joseph A. Buschjbusch@taxonomystrategies.com

http://ww.taxonomystrategies.com