Architectures forExpanded Usability: A
Research Agenda
Sheila O. DennDoctoral Candidate
School of Information and Library ScienceUniversity of North Carolina at Chapel Hill
December 8, 2005
The The GovStat GovStat ProjectProjectFind what you need,understand what you find.
Integration of Data and Interfaces to Enhance HumanUnderstanding of Government Statistics:Toward The Statistical Knowledge Network
Outline
Usability as an evolving concept Software usability Information usability
Information usability research perspectives Metadata Human-computer interfaces
Information usability evaluation An ongoing research agenda
Usability
Defined in various ways, but usuallyconcerned with the following qualities (fromNielsen): Learnability Efficiency Memorability Errors Satisfaction
Usability (cont.)
Traditionally has been a software engineeringconcern
Emphasis largely on software functionality . . . But what of the Web, where the software
(the browser) is the same for eachinformation source, but the content isdifferent?
Information Usability
My definition: the extent to which contentencountered by a user is able to beunderstood and manipulated by that usersuch that she can make correct inferencesabout the meaning of that content
Content InterfaceMetadata User
Represented in Rendered by Presented to
Requests fromQueriesRetrieves
Content InterfaceMetadata User
Represented in Rendered by Presented to
Requests fromQueriesRetrieves
Metadata Research “Integration of Data and Interfaces to
Enhance Human Understanding ofGovernment Statistics: Toward the NationalStatistical Knowledge Network” (NSF GrantEIA 0131824) aka The GovStat Project
“Criminal Justice Statistics IntegratingMetadata Application Profile” (US Bureau ofJustice Statistics Grant 2005-BJ-CX-K016) aka The CJS I-MAP Project
Goals of the GovStat Project
To create an integrated model of user accessto and use of US government statisticalinformation (The Statistical KnowledgeNetwork)
To design and test prototype interface toolsto support finding and using statistics
To support integration (technical andintellectual) of statistical data
GovStat SKN Data FlowAgencies
SKN RegistryActions
Contribute Find
Display Annotate
UnderstandManipulate Collaborate
…..
………….
Objects Actions
Private Work Space
Objects Actions
Private Work Space
Ontology Rules & Constraints
SKN Consortium
…..
Objects Reports metadataTables metadataPeople metadata
GlossaryAnnotations
Objects Actions
Private Work Space
Objects Actions
Private Work Space
Metadata in the GovStatProject Conducted initial user study to investigate how metadata
affects the use of US government statistical websites Modeled metadata based on results of user studies Built initial data model based on DDI and ISO/IEC 11179 Developed a strategy to “test” and further develop the schema Tested mark-up via a scenario Through the markup process, determined that there was too
much complexity in the data model for representing tabulardata; developed a streamlined data model in response
GovStat Metadata User Study
Goals and research questions How do experts and end users undertake
information integration activities? What kinds of metadata do experts and end users
need to complete these activities? What are the implications of what we learn about
these activities on developing tools and interfacesto support integration?
Methodology
Scenario-based design to develop tasks Interviews with 14 expert users, 9 student
users where they were given tasks tocomplete
Think-aloud protocols, follow-up interviews Extraction of themes and stories to illustrate
metadata needs
Results
User Knowledge Surveys and Statistics Interpretation of Information Date Issues Geographic Issues Navigation Content Layout Terminology
Integration Activities Make comparisons across
geographic units definitional differences within concepts and variables units of time different information sources index values
Note discrepancies Manipulate statistics
Integration Challenges Lack of definitions or source information Lack of user knowledge of
Appropriate strategies for using statistics Nature of index values and their use Nature of survey/census purpose and approach Domain and social science statistics
processes/approaches/definitions Interface design problems including interface
inconsistency Inconsistent data across sources User inability to determine if statistics were available Terminology differences across sources
Can Metadata Help? Common user problems that can be addressed
through metadata Mapping of agency terms to user terms Definitions of statistical/survey terms Comparability of statistics Help with finding and interpreting statistics Information about recency of statistics, update schedules,
and propagation of updates to websites Geographic levels at which statistics available Navigation by means other than large lists of text links
Metadata SchemaDevelopment Process Identified and modified/expanded existing elements within
standards (such as the Data Documentation Initiative (DDI)) tofit our needs
Discussed our initial element set and created DTD Used scenarios to choose sample tables for markup Revised based on discussion and samples Held a metadata symposium with metadata experts to help
refine our process and our elements Converted the DTD to an XML schema Revised schema to accommodate aggregate data in a more
streamlined fashion
Current State of the GovStatSchema The current metadata model includes:
Effort to balance complexity with functionality Removal of elements designed to align data values
and row/column headings with survey variables Ability to add on to the model to represent
additional information using a “hierarchy ofintegration” (which will be discussed shortly)
Schema and documentation can be found athttp://ils.unc.edu/govstat/metadata
GovStat Metadata Philosophy
To provide sub-document level access andintegration across documents and agencies.
To provide a minimal set of metadataelements necessary while allowing forextensibility.
To achieve these goals in a manner thatenables efficient transfer to agencies.
Hierarchy of Integration
Low level ofintegration
High level ofintegration
• Searchable table titles
• Searchable row and column headings
• Linking of data values to row andcolumn headings
• Linking of row and column headingsto underlying survey variables
• Linking of analysis units, universestatements, concept definitions, acrossdocuments and agencies
• Linking of contextual information(such as footnotes) to tables,row/column headings, or data values
Our schema canprovide the itemsbeneath this dottedline.
Hierarchy of Integration
Low level ofintegration
High level ofintegration
• Searchable table titles
• Searchable row and column headings
• Linking of data values to row andcolumn headings
• Linking of row and column headingsto underlying survey variables
• Linking of analysis units, universestatements, concept definitions, acrossdocuments and agencies
• Linking of contextual information(such as footnotes) to tables,row/column headings, or data values
Our schema canprovide the itemsbeneath this dottedline.
Organizations can determine wherethey “sit” on the hierarchy in terms ofeffort required and level of integrationdesired.
Goals of the CJS I-MAP Project
Integration and synchronization of criminaljustice statistical information between BJSand its agency partners through thedevelopment of a metadata application profile
Emphasis on facilitating end user discoveryof relevant information items across disparatesources
CJS I-MAP Deliverables A Metadata Application Profile (MAP) to facilitate
metadata sharing and reuse Primarily directed towards supporting users’ discovery of
data Evaluation of the MAP
Technical review User-oriented studies
Final report Methodology for MAP development User studies report
Recommendations for implementing and using theprofile at BJS and partners
CJS I-MAP Project Approaches
Partnering with agencies Theoretical perspectives/approaches
Supporting statistical information integration andthe role of metadata
Usage of metadata: key findings and questions Strategies for building a metadata application
profile
Partnering with Agencies Agency partners include:
Bureau of Justice Statistics (BJS) Federal Bureau of Investigation (FBI) Federal Justice Statistics Resource Center (FJSRC) National Archive of Criminal Justice Data (NACJD) National Center for Juvenile Justice (NCJJ) Office for Juvenile Justice and Delinquency Prevention
(OJJDP) Sourcebook for Criminal Justice Statistics
Partnering with Agencies(cont.) Project goal is to develop a “real-world” solution for
the agencies that enables us to extend the state ofthe art/knowledge in metadata design andimplementation
Through these partnerships we will achieve: Exchange of expertise Ongoing conversations (including some real-time, face-to-
face) Intermediate products/insights on which we will ask for
feedback Joint development of user study research questions
MAP Strategies Identify existing schemas from which we may
be able to draw elements: Data Documentation Initiative (DDI) Federal Enterprise Architecture Data Reference
Model (FEA DRM) Global Justice XML Data Dictionary (GJXDD) GovStat ISO/IEC 11179 Statistical Data and Metadata Exchange (SDMX)
MAP Strategies (cont.)
Identify relevant elements from each schema Identify any necessary elements that cannot
be pulled from other schemas Integrate elements from schemas, fully
documenting any new elements created Process of ongoing iteration with partners Validate with user studies
Future Directions for MetadataResearch Achieve better understanding of trade-offs between
costs of metadata creation and user/agency benefitsin various tasks
Assess hierarchy of integration for Supporting agency decision making Supporting user understanding
Understand extent to which metadata usage isdomain-specific Or when and how can we generalize about metadata
usage, metadata system design
Human-Computer InterfaceResearch
“Toward More Effective Information Use:Fostering Cognitive Integration via theHuman-Computer Interface”
Dissertation project Proposal approved September 2005
Background Access to more information than ever before Effective use of information more difficult Information use requires information
integration What do we know about cognitive process of
information integration? Can we use what we know to improve
information interfaces? What does “improve” mean?
Theoretical Framework Educational psychology
Bloom’s Taxonomy of Educational Objectives (Bloom, etal., 1964) One source of evidence of “understanding” is that learner
can combine information at hand with previous knowledgeto create a new understanding
Cognitive psychology Cognitive Load (Feinberg & Murphy, 2000; Sweller, et al.,
1990) Scaffolding Theory (Clark, 1997) Distributed Cognition (Hollan, et al., 2000; Hutchins, 1995;
Norman, 1993.)
Methodology Extensive literature review to identify promising
interface components Securing agreement to use a particular interface
component (the Hunter Gatherer interface built byschraefel, et al.)
Creation of tasks for user study Pilot testing of experimental interface component
and tasks Revision of tasks as necessary Full testing of experimental interface vs. regular
browser interface
Experimental Interface Hunter Gatherer interface (schraefel et al., 2002) Designed as an integrated module to a Web
browser that allows the user to cut and paste piecesof a web page into a “collection”
Idea was that this could be done within a browserwindow, without the user having to exit the browserto use a text editor
Acts like a combination bookmarking tool and editor
Subject Tasks
Subjects will be recruited through UNCCareer Services
Tasks will require students to imaginethemselves as career consultants forstudents of a particular major, andrecommend one of two cities based onstatistical information that can be found onthe Web (such as at the Bureau of LaborStatistics)
Study Artifacts and Analysis Subjects will be asked to complete a brief pre-study
questionnaire about basic demographic characteristicsand experience with the Web
Subjects will be asked to write a brief statement outliningwhich city they would recommend, annotated with datafound in their search
Subjects will be given a usability questionnaire, askingabout the efficacy of the interface they used and theirsatisfaction with the results
Subject statements will be analyzed by the researcher interms of Bloom’s synthesis and evaluation levels
Experts in UNC Career Services will analyze subjectstatements, and these analyses will be compared withthe researcher’s analyses to ensure inter-rater reliability
Expected Results Results should show if the Hunter Gatherer interface
has any advantage over the standard browserinterface in terms of subjects’ understanding of thedata they find.
If there is a positive effect, then this could be a basisfor recommending the Hunter Gatherer module ininterfaces to complex information.
This research should also demonstrate whether theuse of the concepts of synthesis and evaluation fromBloom’s taxonomy are useful for evaluatinginformation usability.
The Research Agenda Integration is key
In more and more contexts, there is a need for theability to bring together data from disparatesources. Data Integration refers to those activities that occur
on the back-end, such as aggregating selected datafrom distributed sources into a centralized repository.
Information Integration refers to the cognitiveprocesses undertaken by an individual user tosynthesize disparate bits of information intounderstanding.
The Relationship betweenMetadata and Integration
Standard metadata schemas can enhancedata integration by allowing crawlers orother kinds of agents to pull materials fromsources based on the values of metadataelements.
Standard metadata schemas can alsoenhance information integration byallowing users to group data according to thevalues of metadata elements.
The Relationship betweenInterfaces and Integration
Interfaces can be used to provide portals toinformation from disparate sources, or indifferent formats, so that the informationappears integrated from the user’sperspective.
Interfaces can be used to providefunctionality for users to integrate informationfrom disparate sources, or in differentformats, themselves.
Future Directions Explore and describe the relationships between
human cognition and human-information interaction. Explore and describe the relationships between
information architectures and human-informationinterfaces.
Expand the notion of usability to include informationusability. Explore what it means for information to be usable. Explore how different visualizations of information impact
information usability. Move toward greater theoretical foundations for
human-information interaction.
For More Information
Sheila O. [email protected]
http://ils.unc.edu/~denns/http://ils.unc.edu/govstat/
The The GovStat GovStat ProjectProjectFind what you need,understand what you find.
Integration of Data and Interfaces to Enhance HumanUnderstanding of Government Statistics:Toward The Statistical Knowledge Network
Top Related