The agenda…

I. Introduction

II. CROSSTECHONR2 Presentation and DemoDiscussion

III. BlueLiteONR1 Presentation and DemoDiscussion

IV. Wrap-Up

How it began…

Technical manuals are oversized, awkward, hard to move

Navigating through these can be disorienting and confusing

The Navy must maintain and repair a large array of complex on-board machinery.

Our endeavor…

The system we are building will negate the need for paper manuals.

It will use speech and touch input on a Fujitsu Tablet PC, providing flexible access to maintenance information.

After Norfolk, the two ONR groups from the data gathered on the naval base. These two agreed to separately tackle two main needs that emerged needs were navigation and collaboration. In the last three months, these two groups have prototyped, researched, and iteratively designed multi-modal IETM systems with specific emphases in these areas.

THE IETM KNOWLEDGE BASE

CROSSTECH

Agenda

1. Why Collaboration2. Collaboration Evolution3. Design Cycle4. CrossTech Demo5. User Studies6. Case Study7. Tech Rationale

Display ConsoleSLQ-32 v3

Equipment Assembly Cabinet

Our Vision

Buiild a portable IETM that supports:

Collaboration with annotation

Pen + speech I/O

Expert Knowledge Sharing

Placeholding

The Collaboration Evolution

KnowledgeSharing Tool

Non-CollaborativePreventative Maintenance

Support System

Same ship / Different deck

Collaboration Tool

Troubleshooting ScenarioNaval Signal Select Test

Norfolk User Research

Knowledge Sharing Tool How is this a Knowledge sharing tool?

Supports two types of tasks:• Troubleshooting: supports investigative nature of

the task. Users may seek outside experts for help. • Still supports the PMS tasks where navigation is

often rigid and sequential

Supports two types of Collaboration:• Collaborating synchronously with a partner

working on the same task• Sharing Knowledge about the SLQ32 with experts

off-shore

Spring Design CycleNine main prototypes before the summer:

Extreme Design Sessions

Norfolk

Spring Finals

Summer User Study/ Design Cycle

In the summer, we began with two different designs :

After Phase 1 of the summer user studies, we consolidated the best of these two prototypes into one as the basis for our current design.

CROSSTECH

A knowledge sharing system that serves as a centralized point where IETM information and expert knowledge is exchanged and stored

CROSS: Derived from “crosshair” and “cross collaboration”

TECH: derived from Tech manuals and “techs” as the jargon for naval mechanics

S Y S T E M D E M O

Login

1. All naval personnel have identification cards (can further support fingerprint ID technology)2. Client Request

RATIONALE

Task Assignment List

1. Cannot find problem in the manual [CI-NAVY1-b2]2. Flip through manuals for a long time[CI-NAVY1-b1]

RATIONALE

N A V I G A T I O N

NAVIGATIONAL ISSUES

Search through the tasks and easily branch off into sub-tasks.

Efficient navigation through steps and diagrams.

Keep track of where they have been and store temporary references.

Keep the workspace manageable

Global Navigation

1. Users use diagrams to remind themselves of the steps taken [CI4-HUNT2-o1]2. Loses place in the instructions [CI10-AR-b1]3. User scans through all of task before actually beginning task [CI10-AR]

1. Users have a hard time clicking on the global navigation and end up not using it (made it bigger) - [USp2-3]2. Users now prefer to use the global nav to navigate locally -P33. Other users still hesitant to use global nav, so added a hover-over effect - P34. Having an outline around the global step they were on did not provide enough feedback (changed it to a solid color) - P25. Navy Users like the overview to see the high-level [USp0-3]6. Users prefer to see icons denoting warnings, steps, cautions and diagrams so they know what step to go back to.

RATIONALE

USER FEEDBACK

Bubble Steps

1. Users use diagrams to remind themselves of the steps taken [CI4-HUNT2-o1]2. Gather all the tools at once [CI9-NAVY1-048]

1. Users do not notice warnings, notes and cautions when they are included in steps (need to add icons in global and colors and icons in bubble) - [USp1-5]2. Users get confused at the beginning and end of the task when there are up and down arrows (Remove arrows at beginning and end) - P23.Underlined Diagram is not seen as clickable to the user. (USp1-13)4 Users have problems navigating through steps(USp1-16)5. Wont do anything until they see a diagram

RATIONALE

USER FEEDBACK

Shared Space / My Space

1. One works on Console and equipment while other works ont he manual [CI1-NAVY-o1] == myspace shared space2. Students try to be on the same page in the manual and flip through it a long itme [CI6-PIA-b1]3. Cannot see what step the other person is on in the MRC Card [ Ci9-NAVY1-b4]4. Collaborators run back and for to see what other sees [CI9-NAVY1-o12] 5. shared view would be helpful [CI9-NAVY1-o13]

1. Users lose a sense of state - they inadvertently move into SharedSpace and do not have feedback to tell them what happened. P1, P22. Changed the colors on tabs - P23. Changed the background texture of the different spaces - P34. Windows-style icons are preferred by the Navy users [USp0-5]5. Having an import arrow is confusing to users when in the help space (so we removed it).6. People do not know how to easily get to and from shared space (USp1-7)

RATIONALE

USER FEEDBACK

Import Step / Diagram

1. Print out Diagrams for reference 2. Cannot find diagram in manual (flipping through manual for 15 minutes) [CI5-PIA1-o5]3. Students rip out pages from manual to use [CI5-PIA1-b7]4. Students use screwdrivers and pens to hold pages in the manual [CI5-PIA-o7]5. Bookmarks needed only for current task(don't need to be saved) [CI9-NAVY1-o62]

1. Links to open Diagrams are not noticeable - users not clicking on them (made them more noticeable) - P22. New thumbnails make the user think that that is the entire diagram they need to reference - P33. Changed the design to a icon and a blue Diagram open button - P3

RATIONALE

USER FEEDBACK

Show / Hide Steps

1. Client Feedback2. Table Students work on is cluttered [CI5-PIA1]

RATIONALE

USER FEEDBACK1. Users have difficult time targeting the function button - Phase 3

Task Branching1. Scenario provided to us from Navy has multiple subtasksRATIONALE

USER FEEDBACK1. Uses say that having only the number on the task branching tab does not provide enough meaning to them - Phase 3

Search

1. The Project team is also working closely with Pearson KAT, so we needed to prototype this feature.

1. Users often default to entering text in the search box when they are lost in the task - P0

RATIONALE

USER FEEDBACK

S P E E C H

SPEECH ISSUES

• Multimodality is useful for eye-busy, hand-busy situation

• Valuable for mutual disambiguation

• Speech acts as an accelerator- makes the task easier

Speech

USER FEEDBACK

1. Hands busy most of the time [CI9-NAVY1-o25]2. Environment is loud, cramped and small. Always Make speech robust [CI1-NAVY1-b4]3. Place Manual on closet table with relevant page left open [CI1-NAVY1-o2]

1. Users have no visual feedback of what they said or what happened (incorporated a text window of what they said) - P22. One word grammars get misrecoed a lot when equipment or parts are being used (switched to multi-word grammars) - P23. Not enough feedback of when they are in the speech system (added orange highlight) USp1-64. Users often forget the active grammars - P2

RATIONALE

C O L L A B O R A T I O N

Display ConsoleSLQ-32 v3

Equipment Assembly Cabinet

COLLABORATION ISSUES2 person collaborative task

Work and communicate with non-collocated users.

Need a private workspace during collaboration session, in case users need to search for information for their own use

A picture is worth a thousand words – good share visual condition of the task

Voice recognition interferes with shared phone capability

Phone

1. "We dont know whats going on" - studnest waiting for students inside the craft [CI6-PIA2-b2]2. Delayed response, cannot hear answers clearly [CI6-PIA2-b6]3. Maintenance sites have sound powered phones that do not always work [CI9-NAVY1-84]4. Don't know what person is doing without direct communication [CI9-NAVY1-o16]

1. Not enough feedback of when phone is on or off (added orange highlight)- USp1-42. Not clear how to active the phone connection (added grammar to label) - P2

RATIONALE

USER FEEDBACK

Annotations

1. Students use color to mark up diagrmas showing the differnet features [CI5-PIA-o5]2. Sharing picture, annotations, video would be helpful for collaborators [CI9-NAVY1-o7]

1. Screen gets cluttered with annotations very quickly, added eraser tool - [USp1-19]2. One point line is not thick enough to annotate (changed to four point line) - P3

RATIONALE

USER FEEDBACK

Camera

1. Sharing picture, annotations, video would be helpful for collaborators [CI9-NAVY1-o7]2. Collaborators run back and for to see what other sees [CI9-NAVY1-o12] 3. Pictures of equipment would be helpful, dont always know what it looks like [CI9-NAVY-o50]

1. All functions and icons that deal with sharing should be semantically grouped together - P12. Camera Icon provided a clear affordance of what the feature does to users - P2

RATIONALE

USER FEEDBACK

K N O W L E D G E S H A R I N G

KNOWLEDGE SHARING ISSUES

Have access to expert information when experts are not readily available

Some maintenance problems need advice from experts who are not located on site

Colleagues might have done a particular task before. The notes they left could be useful for workers doing the same task.

Tips

1. Car Mechanics like to "tear into" system [CI2-CAR-02] -- give tips to give help when needed, and not dictate their terminology in the manual difficult to understand.

2. Novice user receives tips and advice from the expert and tires to learn from the experience [CI4-HUNT2-o2] 3. Updates to manual also come in form of comic book which also includes tips for maintenance [CI4-HUNT2-o4]

4. Students ask instructors for help first [CI5-PIA1-o3]

RATIONALE

Suggested Content

1. Students rely on instructors when faced wiht a problem instead of troubleshooting using the manual [CI6-PIA2-o2]2. User needs additional help [CI-10-AR-b4]3. Most frequently used content should bubble up to the top [Faculty Feedback]

RATIONALE

USER FEEDBACK“Help” tab is confusing. User thinks it is system help- Phase 3

Suggested Contacts

1. Requesting Help via sending messages/recieving messages by sending pictures2. Students ask instructors for help first [CI5-PIA1-o3]3. Pictures of equipment would be helpful, dont always know what it looks like [CI9-NAVY-o50]

RATIONALE

U S E R S T U D I E S

We ran nine Naval technicians and civilian contractors through four paper prototypes -- gathered issues related to usability and usefulness.

We ran eight members of the CMU community through a combined paper-prototype and digital collaborative tools (Microsoft Netmeeting).

PHASE ZERO - close encounters with naval kind

PHASE ONE - what features did we want to keep?

User Study Phases

We ran five users through our entire prototype, and made quick iterative changes and noted larger issues we were going to take of before phase three.

PHASE TWO:Basic interaction/usability issues

PHASE THREE:Validating our design

User Study Phases

We tested another five users on the larger issues we corrected from phase two, and noted outstanding issues.

Task Simulation : Antenna RotatorADVANTAGES

– It matches well to the Navy scenario– Manual Format Similar to IETMs– Scalable task (30m - 1.5h)– It incorporates hands-on mechanical

work– Hands busy task– Could test people without mechanical

repair experience since it was meant to be assembled at home

Implications of the Antenna Rotator Task

GOOD DATA FROM THE TASK

Easy to recognize user errors while doing the task since we knew the correct sequence intimately

Users used speech abundantly as the task required physical manuveuring and they were hands-busy

Simulated 2 person task with one person outside assembling antenna and the person inside wiring the rotator

Base

Antenna

Coaxial Cable

NOT USEFUL DATA

Bad feedback regarding the language in the manual

Difficulty with wiring when testing with less skilled users

Whack-a-Mole: A Case-study in Usability

1. Initially we used a link to denote where the diagram was in the steps. Users were reading through the steps and completely bypassing the links. This led to many errors in their task.

Whack-a-Mole: A Case-study in Usability

2. To provide a better affordance, we changed the link to a button. This worked well on most steps, however when placed next to a ‘Warning’ note, users again did not notice the button, perhaps because of the similar color tones.

Whack-a-Mole: A Case-study in Usability

3. We then decided to put a thumbnail of the diagram in the step, thinking that the users would associate the button with the thumbnail. Instead, they would just try to read from the diagram without clicking on the Diagram button.

Whack-a-Mole: A Case-study in Usability

4. In our final iteration, we removed the thumbnail but added in icons that denoted figures, tables, and other steps. We also changed the color to blue so that it is consistent with the blue headings of the figures and tables.

T E C H R A T I O N A L E

Tech decisionsLimitations of technology also serves as a sanity check.

– Don’t want a design that can’t be built!

JavaScript – Language works with all speech tools– deployable on a browser which is the current

technology used by clients and other collaborators

Microsoft Speech – A robust commercial speech engine– can be integrated easily with JavaScript– Easier development (can add grammars)– Can use it with Internet Explorer

We learned during our trip to Norfolk that the Navy technicians need to bring an Oscilloscope and a Multimeter to the majority of their daily tasks:

Naval technicians confirmed that a Tablet PC would be a good size considering they need to bring these tools with them as well.

Why a Tablet PC?

AcknowledgementsBob KrautCarolyn RoseBonnie John

Jane SiegelAlex RudnickyEllen AyoobBrian GollumYitao SunThe Office of Naval Research

User Feedback: Sarah, Kevin All our other users!

Our lovelies in ONR1:HariJanetteJennJoonPoloSandhya

Peer Crit Group:

JeffJessicaLorriMarina

C O D E W A L K T H R O U G H

The two teams have agreed that the following are the eight most important observations/needs that should be explored in the future:

1. Simulation: Integrating existing tools (o-scope and the multimeter) into the interface

2. Explore and gather user data regarding knowledge sharing in the context of the Navy

3. Speech error input design4. Auto-logging during tasks5. Saving/history of annotation6. Development of user profiles7. How to enforce reading warnings, cautions, and

notes.8. Integrating SuperManual results into suggested

content

Feature StrengthsONR 1• Diagram Zoom/Pan• Drill down diagrams• Grammar • Multimodal input/output flexibilty• Signal Tracing• Managing multiple information sources

ONR2

• Window/space management• Collaborative spaces spaces • Global overview nav bar• Resource/Expert Content• Placeholding• Camera interaction

BOTH GROUPS• The design of steps: how they are handled and displayed• Task branching