A Human Factors Evaluation of Illinois Compass Presented at the CITES Educational Technologies Brown...
-
Upload
lilian-stokes -
Category
Documents
-
view
213 -
download
0
Transcript of A Human Factors Evaluation of Illinois Compass Presented at the CITES Educational Technologies Brown...
A Human Factors Evaluation of Illinois Compass
Presented at the CITES Educational Technologies Brown Bagby Esa M. Rantanen, Ph.D., on April 13, 2005
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
• What is Human Factors?
• Human-Centered Design philosophy.
• Barriers to Human-Centered Design.
• Human Factors in systems’ development.
• Human Factors evaluation of the Illinois Compass.
• Results: Suggestions for improvement.
• Conclusion.
Presentation Outline
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human Factors: Definitions
• “Human Factors is that branch of science and technology that includes what is known and theorized of about human behavioral and biological characteristics that can be validly applied to the specification, design, evaluation, operation, and maintenance of products and systems to enhance safe, effective, and satisfying use by individuals, groups, and organizations.”
(Christensen, Topmiller, & Gill, 1988).
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human Factors: Definitions
• “Human Factors is that branch of science and technology that includes what is known and theorized of about human behavioral and biological characteristics that can be validly applied to the specification, design, evaluation, operation, and maintenance of products and systems to enhance safe, effective, and satisfying use by individuals, groups, and organizations.”
(Christensen, Topmiller, & Gill, 1988).
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human Factors: Definitions
• Human Factors studies human capabilities and limitations as they apply to the design of systems and products.
• Ergonomics (from gk. ergon ‘work’ + nomics) is the application of scientific information concerning humans to the design of objects, systems and environment for human use.
• The goal of human factors is to…– …reduce error– …increase productivity– …enhance safety, and– …enhance comfort
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human (or User-) Centered Design
• A design philosophy, or methodological principle, that centers the design process around the user.
• Three important attributes:1. Focus on the roles of humans in complex systems2. Design objectives are elaborated in terms of roles of humans3. Specific design issues follow from these objectives
• Note: No explicit focus on performance, productivity, economics, or related metrics; they subserve human-centered design issues because they are the consequence of good design rather than means for creating the design.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human (or User-) Centered Design
• Also known as Usability Engineering.
• Four general approaches to design:1. Early focus on the user and tasks2. Empirical measurement using questionnaires, usability
studies, and usage studies focusing on quantitative performance data
3. Iterative design using prototypes, where rapid changes are made to the interface design
4. Participatory design where users are directly involved as part of the design team
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human-Centered Design: Background
• Systems have increased in size, scale, and complexity, as economic and technological necessities mandate increased performance
People will increasingly become like “cogs of machines.”
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human-Centered Design: Background
• However, automation can never be legally, ethically, and socially responsible for its actions
• Hence, regardless of systems’ scale and sophistication, humans will always have the ultimate responsibility of their operation.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human-Centered Design: Background
• Because people will inevitably be responsible for system operation, they must…
1. …perceive the nature of these responsibilities, and
2. …have appropriate levels of authority to fulfill them.
• That is, people have to be “in charge.”
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human-Centered Design Objectives
• Design objectives should be to support humans to achieve the operational objectives for which they are responsible
• Hence, the purpose of a pilot is – not to fly the airplane that takes people from point A to point B;– instead, the purpose of the airplane is to support the pilot, whose
responsibility is to take people from A to B.
• The purpose of factory workers is – not to staff a factory designed to achieve some productivity goals;– instead, the purpose of the factory is to support the workers who are
responsible for achieving productivity goals.
• What is the purpose of the Illinois Compass?
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human-Centered Design
• Objectives: Human centered design should…1. …enhance human abilities2. …help overcome human limitations3. …foster user acceptance
• Design Issues: 1. Formulate the right problem -- make sure that system objectives
and requirements are right2. Design an appropriate solution -- excellence in engineering is
necessary but not sufficient to assure that system design is successful
3. Develop the solution to perform well -- operability, maintainability, supportability
4. Assure user satisfaction
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Barriers to Human-Centered Design
• Miscommunication– For example: A user “…wanted an operating system that
was stable, up-to-date, and easy to use, one that would enable me to get my work done effectively. It is not clear that my goals were shared by those who created the installation process for the OS upgrade. Instead, they seem to have focused on creating an appealing interface; a simple, automatic installation process; and a means of taking every opportunity to inform the user of all the great system features.”
Alan Hedge in Ergonomics in Design, 11(3), 4-9
• Technology-driven design
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Miscommunication in Design
As proposed by the project sponsor
As specified in the project request
As designed by the senior analyst
As produced by the programmers
As installed at the user's site
What the user wanted
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Technology-Driven DesignPerformance/productivity
requirementsPerformance/productivity
requirementsTechnological “opportunities”Technological “opportunities”
Designers infuse technology
Designers infuse technology
Maintainers overwhelmed by technology
Maintainers overwhelmed by technology
Managers overwhelmed by Data
Managers overwhelmed by Data
Operators overwhelmed by alternatives
Operators overwhelmed by alternatives
Designers produce technology ‘fixes’
Designers produce technology ‘fixes’
Increased system complexity
Increased system complexity
Increased mis- and disuse of system
Increased mis- and disuse of system
Increased automationIncreased automation
Performance/ productivity shortfallPerformance/ productivity shortfall
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human Factors in Systems’ Development
Operational RequirementsOperational
RequirementsFunctional
RequirementsFunctional
RequirementsSystem
FunctionsSystem
FunctionsFunctional
LogicFunctional
LogicUser
InterfaceUser
Interface
Marketing,Customers
Engineering, HumanFactors
Human Factors
User goals, existing mental models, task- and environment analyses
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Systems Development Life Cycle
• Stage 1: Front-End Analysis• Stage 2: Conceptual Design• Stage 3: Iterative Design and Testing• Stage 4: Design of Support Materials• Stage 5: System Production• Stage 6: Implementation and Evaluation• Stage 7: System Operation and Maintenance• Stage 8: System Disposal
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
…and Associated Human Factors Activities
• Stage 1: Front-End Analysis– User Analysis– Preliminary Task Analysis– Environment Analysis– Identification of User Preferences and Requirements– Input for System Specifications– Make sure objectives and functions match user requirements– Provide ergonomic criteria
• Stage 2: Conceptual Design– Function Allocation– Support for the Conceptual Design Process
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
…and Associated Human Factors Activities
• Stage 3: Iterative Design and Testing– Task Analysis– Interface Design– Prototype Development– Heuristic Evaluation (Design Review)– Cost-Benefit Analyses– Trade-off Analyses– Workload Analysis– Simulations and Modeling– Safety Analysis– Usability Testing
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
…and Associated Human Factors Activities
• Stage 4: Design of Support Materials– Develop and provide input for support materials
• Stage 5: System Production
• Stage 6: Implementation and Evaluation– Evaluate Fielded System
• Stage 7: System Operation and Maintenance– Monitor System Performance Over Time
• Stage 8: System Disposal
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Task Analysis: Definitions
• A collection of techniques for describing how people interact with systems.
• A way of systematically describing a system to better understand how to match the demands of the system to human capabilities
• Function analysis is similar to task analysis except that it focuses on basic functions that will be performed by the human-machine system. – A function is generally composed of many tasks.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Task Analysis
• Task analysis seeks to identify…– User goals and their associated activities
– Tasks and subtasks necessary to achieve the goal
– Conditions under which tasks are performed
– Outcomes of the tasks and subtasks
– Information or knowledge needed to perform the tasks and subtasks
– Communication with others for performing the tasks
– Equipment needed to perform the tasks
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Task Analysis (cont’d)
…for– Training requirements– Interface requirements– Process redesign– System reliability– Staffing requirements– Workload estimation
…using methods for– Data collection– Data representation (lists, outlines, and matrices)– Data analysis and modeling– Observable (physical) and unobservable (cognitive) activity
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Task Analysis: Examples
• General methods for data collection– Document and equipment analysis– Unstructured and structured interviews– Group interviews and focus groups– Sorting and rating– Observation– Verbal protocol analysis– Questionnaires
• General methods for data representation– Lists and outlines– Matrices (cross-tabulation tables)– Structural and hierarchical networks– Flow- and time-line charts
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Task Analysis: Examples
• Specific task analysis methods– Controls and displays analysis– Hierarchical Task Analysis (HTA)– The GOMS model– Critical incident technique– Conceptual Graph Analysis– Activity Sampling– Operational Sequence Diagram
– Cognitive Task Analysis (CTA)
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Usability
• User Interface– How the product looks to the user?
• Functional Logic– How the product does what it does?
• Functions– What the product does.
…All should be transparent to the user!
Provide a good conceptual model Make things visible Use natural mappings Provide feedback
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Usability Testing
• Definition (and quantification!) of usability– Learnability– Efficiency– Memorability– Errors– Satisfaction
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Summary
•‘Know• Thy • User’
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human Factors Evaluation of Compass
• Evaluation of Compass was carried out as a classroom project in the AVI/PSYC 358/IE 340 ‘Human Factors’ course
• Students were encouraged to supply anecdotal usability data by posting their experiences to a Compass discussion forum
• A group of students evaluated Compass and suggested areas for redesign as a semester project.
• Despite the hours expended by participating students, this evaluation was far from thorough and only anecdotal data could be gathered.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Human Factors Issues of Compass
• Compass is a feature-rich system and consequently quite complex
• Complexity results in– Increased learning time and training needs,
particularly with large and constant proportion of novice users
– Under- or disuse of many potentially useful features– Decrease in user acceptance and satisfaction
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Selected User Comments
• ‘I used the address book to send e-mail to the class. At first, I thought "save" meant that it would save my selections (as it does in Express webmail), but it displayed the addresses in my "To" box. Because of this, I assumed that I could check any boxes on any page of the address book, and it would all show up as when I hit "Save". I found out that it only does this for one page at a time.’
• ‘There isn't a link to the Illinois Compass calendar from the web page of each course. The only link to the calendar is available at the Illinois Compass Home page. Therefore, if you are in a course home page and you want to use the calendar you have to return to the Illinois Compass Home page each time you want to refer to the calendar.‘
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Selected User Comments
• ‘On the main page of compass, it only notifies you of an update if there is a new email, new discussion post, or new grade (there might be more). So if an instructor for the class adds a new assignment or changes anything inside the class subsection, you won't know unless you explore it. This …means you have to search through each class if you want to make sure you haven't missed something. And if you don't have time, you might miss something important.’
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Selected User Comments
• ‘This happens every time I use the discussion board. Whenever I click on a post that I want to read, it opens it up in a new window. Once it opens, it makes me re-login to compass in the new window. This then brings me back to the main compass page where I have to find the post that I wanted to look at. Once I click it, it opens yet another new window with the post, not requiring me to re-login this time. Essentially I end up with three compass windows. I'm not sure if this is because of my browser, or some setting within my browser. Perhaps they should just have posts viewable without the need open a completely new window.’
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Selected User Comments
• ‘Since a lot of my classes have used compass previously, I have a cluttered up list where I have to search each time I want to find the class I'm looking for. Perhaps it would be even better if we were able to sort the list of classes in some way.’
• ‘When teaching assistants are in a "student view" screen, the user-friendly link to the Illinois Compass Home does not appear on the top right of the screen, which makes getting back to the Home confusing. Ultimately, one must remember to go back to the "build" view for the link.’
• ‘When trying to save lecture notes, you cannot right click and "save target as...". Thus, if your browser is configured to automatically open the powerpoint slides in Explorer, you are not able to save the file or change printing options.’
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Selected User Comments
• ‘I had trouble figuring out how to forward the mail from Compass into my other account. When I got into the message center, I was looking for text similar to "E-mail options" that would allow me to forward my mail. I couldn't find anything like that, so I started browsing the page, looking at the options under the dropdown menus, and clicking the "Options” button, unable to find that option. I eventually used the "Search this tool:” to find out how to forward my mail.’
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Selected User Comments
• ‘Yet another fault of the Illinois Compass system is its poor label design. As denoted by Wickens et. al in Ch. 8 (p.193-4), effective labels are: visible, discriminable, meaningful, and have proper location. In this courseware, however, few labels contain all/most of these features. For instance, why are the important labels of: grades, assignments, and discussions clumped together (lack of discriminability) and hidden under “More Tools” (not visible) whereas “Local Content,” which leads to nowhere if clicked (meaningless), has its own label in plane site on the page?’
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
General Human Factors Issues
• No link from the UIUC home page; alternatives:– Type the url: https://compass-portal.cites.uiuc.edu/login.html– Search for Illinois Compass
• Incompatibility with browsers; MS Internet Explorer seems to have lots of problems, yet it is the most popular browser (84%)
• Inconsistent functional logic between Compass and web browsers; yet, users expect browser-like controls
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Specific Issues: Slowness
• Because Illinois Compass has been implemented using the Java programming language, slow loading time can be expected.
• However, this is heavily dependent on the hardware of the user (e.g., amount of RAM, processor speed), specific browser (IE, Netscape, Firefox, Safari), and internet connection (e.g., dial-up modem, cable).
• An increase in user connections to Illinois Compass causes bottlenecks in the network, which also degrades performance.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Slowness: Example
• Using a dial-up modem:– Time to load home page (from authentication): 1 min 40 s– Time to load a course home page: 3 min 05 s– Time to go to ‘Teach’ page/mode: 1 min 29 s– Time to bring up Grade Book: 3 min 28 s– Time to remove a student and return to main: 3 min 20 s– Time to return to ‘Build’ page: 1 min 17 s– Time to go to ‘Lecture Notes’ folder: 1 min 27 s– Time to upload a PowerPoint file (248 KB): 6 min 12 s– Total 21 min 58 s
• Most of the above times were spent on waiting for pages to load and reload.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Specific Issues: Navigation Problems
• Normal browser functions do not work with the Java-based application, e.g., back-button, shortcut keys; this is contrary to user expectations and leads to recurrent errors.
• Button labels are vague or misleading– Link ‘Illinois Compass Home’ goes to class list; the flower icon links
to CITES page about Compass, not to Compass home page
• Button locations are non-intuitive and inconsistent • Lack of a site map• Even simple actions require navigation through multiple pages.• Slowness of the system is exacerbated by clumsy navigation,
requiring reloading pages after majority of actions.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Specific Issues: Grade Book
• The grade formula cannot be easily edited; adding new graded work often requires retyping the entire formula
• Many useful features from the old electronic gradebook did not make it to Compass
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
E-mail Forwarding and Replying
• Received:
From: Student [AVI 497 - Special Topics in Aviation - Spring 2005 - Section ER]To: Rantanen, Esa CC: BCC: Subject: Class Sent: Wednesday, March 30, 2005
• Reply:
• E-mail forwarding must be set separately for each class student is taking
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Suggestions for Improvement
• Overall goal: Reduce navigation through multiple pages when performing simple actions
• Specific suggestions:1. Place Compass website login link on the University website,
www.uiuc.edu The link should take a student or faculty member directly to the login page to save time clicking “Login” multiple times.
2. Clearly differentiate links to Compass home page (class list) and specific course home page with consistent placing of the links.
3. Use of drop-down boxes containing course-specific “requests”, ordered on the left side of the toolbar, from left to right.
4. Add a “GO” button to the right of the drop-down boxes to allow for correction of errors.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Conclusion: Comprehensive Usability Analysis
• A comprehensive usability analysis is essential for successful implementation of human factors in future upgrades of the Illinois Compass.
• However, such analysis is complicated by the number of different computer platforms (PC and Mac), operating systems (versions of Windows and MAC OS), and basic performance characteristics of these (memory, processor speed).
• Furthermore, the complexity of Compass itself makes human factors analysis of the current design and contributions to redesigns very difficult, labor-intensive, and time-consuming.
• Yet, there is no alternative to thorough usability analysis and testing to ensure successful implementation and operation of the system.
CITES Educational Technologies Brown BagApril 8, 2005
viationHuman Factors
Thank You!• Thanks are also due to Shanqing Yin, Jordan Clark,
Patrick Brennan, and Kevin Olson for their term project on Compass usability and to the students of AVI/PSYC 358/IE 340 during the fall semester 2004 and of AVI 447/PSYC 457 during the spring semester 2005 for their thoughtful comments and critique.
Questions?