Usability Testing Project 1

Running Head: USABILITY TESTING PROJECT: MICROSOFT WORD 2007

Usability Testing Project: Microsoft Word 2007

COSC 605 Human Computer Interaction

Towson University

Rebecca Bayeck

Keith W. Beatty

Amanda Dowd

Ebonie Greene

Kyoungha Kim

Usability Testing Project 2

Abstract

This experiment was an attempt to infer a relationship between the claimed familiarity of

a test subject using MS Word and the ability to efficiently complete a series of Word-related

tasks based on the features of Word’s Developer ribbon. A questionnaire designed to rank order

subject experience and familiarity with MS word was created. The null hypothesis (that the

greater the experience with MS Word the greater the likelihood of completing the assigned task)

was not proven. Familiarity with MS Word did not appear to be a factor in the subjects meeting

the hypothesized performance level. Results indicate that participant anxiety and lack of

familiarity with the instruction set were unanticipated factors effecting subject performance.

Additional lessons regarding test setup, tear-down/reset and conduct were valuable to the test

team but served to inhibit the test results.

Usability Testing Project 3

Description of the interface evaluated

Microsoft Word 2007 is a part of the Microsoft Office 2007 Suite which is available for

Windows and Mac OS operating systems. It was developed by Microsoft Corporation as a word-

processing application to be used for typing documentations. The program incorporates various

other aspects of word processing. The application allows the user to format reference lists (figure

2), mailing lists (figure 3), internal document comments (figure 4), and web pages (figure 5).

Figure 1 shows the basic layout of Microsoft Word 2007.

Figure 1: Screenshot of Microsoft Word 2007

The Microsoft Word 2007 interface uses a series of menus to catalogue its vast amount of

features and options. These menus are organized on a platform called “The Ribbon” (“Getting

Started”, 2012). The ribbon is designed to showcase the most popular options and functions for

Usability Testing Project 4

ease of use (“Getting Started”, 2012). The default menus include the Home, Insert, Page Layout,

References, Mailings, Review, and View menus. Some menu options are hidden from users and

may require specific procedures in order to activate and use them. Files are saved under specific

file extensions that are indicative to the type of document created. Microsoft Word 2007 also

allows users to save their documents in the file format of previous versions of Microsoft Word at

the risk of creating compatibility issues. The application also utilizes a help menu to assist users,

which is capable of either referring to assistance information from the program’s local user

manual or the Microsoft support website.

Figure 2: References Menu, Microsoft Word 2007

Figure 3: Mailings Menu, Microsoft Word 2007

Figure 4: Review Menu, Microsoft Word 2007

Figure 5: Developer Menu, Microsoft Word 2007

Usability Testing Project 5

Typical Tasks

As a word processing application, Microsoft Word 2007 is primarily used for writing

documents. Such documents include reports, letters, resumes, brochures, business cards,

envelopes and various others. The application offers a myriad of features that users can utilize to

modify their documents (“Getting Started”, 2012). These features include text formatting, adding

graphics, orienting page margins, adding footnotes and citations, and many more (“Word 2007

Training Courses”). Users of Microsoft Word 2007 typically have the goal of creating one of the

aforementioned documents and either saving, printing, modifying, sending or publishing their

creations. In the context of this project, the aspect of the interface deemed to be of particular

interest was a user’s ability to exploit features in the Developer Menu of Word to create a web

page.

Typical Users

MS Word 2007 has a variety of users. The test was run on MS Word 2007 part of

Microsoft Office Standard edition. This edition is aimed at the typical business user; according to

Forrester Research, as of May 2010, Microsoft Office 2007 was used in 81% of enterprises it

surveyed (its sample comprising 115 North American and European enterprise and SMB

decision makers)1. However, students, teachers, and home users can also be considered typical

users of this application. The age of the users varies, as well as their computing experience.

Beginners or individuals with little or no computing background use this application, as Word

has become the standard when it comes to use computer to write letters. Users with great

computing experience are less challenged when using this application compared to individuals

with less or no computing experience.

1 http://en.wikipedia.org/wiki/Microsoft_Office_2007 (retrieved 18may2012).

Usability Testing Project 6

MS Word 2007 can be used at home, in offices, in classrooms, or libraries. There is no specific

environment for the use of this application as long as the user has access to the application, either

from a public or private laptop, or desktop. Though children as young as four play around with

MS Word 2007, for a good production, the users need to be able to read and write.

A survey of the Microsoft Support Knowledge Base and MSDN Library articles2 would appear

to indicate that a typical user of the MS Word 2007 webpage feature may link such a web-

enabled document to a blog posting. As an upgrade from Word 2003, 2007 introduced new,

smaller file formats, improved document security and integrated with a greater number of

applications than di 2003.

As a feature of Word, the ability to create a new web document could then be seen, in

part, as a somewhat limited attempt to promote the use of Word as a general-purpose web

development platform outside of Microsoft’s proprietary ASP.NET 2.0 environment.

For purposes of this testing, the academic intent was to extract some understanding of Word's

viability as a platform for building web pages. Given that Microsoft embedded web page

functionality into Word as part of Office 2007, it did so for a reason or multiple reasons.

Unfortunately, as a standalone feature, it remains unclear as to whether this functionality of

Word 2007 was ever properly promoted from a marketing standpoint.

How was the task list developed?

Our lab asked participants to create a web page in Microsoft Word 2007. The procedures

were created in effort to expose the participants to as many functionalities, features, and facets of

Microsoft Word as possible, while assisting their web page creation. Firstly, the instructions

touch on the file saving options. Since Microsoft Word 2007 allows the user to save a document

2 http://social.msdn.microsoft.com/search/en-us?query=create+webpage+%2B+Word+2007

(retrieved 17May2012).

Usability Testing Project 7

in various file formats, we first asked our participants to save their document in a web page

(.htm, .html) file format. This is included in order to specifically identify the user’s document as

a web page, which will allow the user to make use of Word’s web page features. The file saving

for web pages gives the user the option to modify the title of the web page, which may be a

helpful feature to users.

As previously mentioned, Microsoft Word 2007 utilizes a ribbon interface, which is

supposed to facilitate the user’s experience. The remaining procedures asked users to navigate

multiple sections of the ribbon in order to assess the ribbon’s true ease of use. Each menu within

the ribbon has its own submenus which bundle similar functions together. The instructions

detailed the particular menus and submenus for the participants to locate the specified functions.

This was done to aid the participants in their attempts to find each feature in the event that they

are confused by the wide array of options. We also gave our users repetitive actions to carry out

during the lab process in order to determine whether they found it easy to remember where to

locate menu features. Participants were asked to use elements within the Home menu to modify

text, the Page Layout menu to adjust the document’s background, the Insert menu to add a

graphic to the web page, and the Developer menu to utilize web page functions.

The Developer menu tab is not a default tab shown in the ribbon. It is used to write and

run macros, use XML commands, use ActiveX controls, and many other technical tasks. The

Developer menu tab also houses the web page functions for Microsoft Word 2007. In the

previous version of Microsoft Word, or Microsoft Word 2003, the HTML functions are

accessible through the Insert menu (“Insert a Drop-down List”, 2012). In Microsoft Word 2007,

the features are only accessible through the Developer menu, which is one of the hidden menus

that the user must access through specific means. To activate the Developer menu, the user must

Usability Testing Project 8

click the Office button, which is found in the ribbon, and select Word Options (“Show Developer

Tab”, 2012).

In the Word Options menu, under the popular tab, the user can select the option to show

the Developer tab in the ribbon (“Show Developer Tab”, 2012). We purposefully left out any

detailed instructions as to how to activate the Developer tab out of the instruction set and

expected the participants to inquire about its location. We were also planning to assess the

participants’ satisfaction with the process of retrieving the Developer menu tab.

Figure 6: Legacy Tools Section of Developer Menu

Within the Developer menu, the web page functions are organized into a category called

“Legacy Tools” (see figure 6). The Legacy Forms section at the top of the Legacy Tools

category includes web page features available in Microsoft Word 2003. The lower portion of the

submenu consists of ActiveX controls. We asked our participants to only use the Legacy Forms

features instead of the ActiveX Controls because the ActiveX Controls options require the use of

Microsoft Visual Studio, which would complicate our usability testing. Microsoft Word 2007

warns that the ActiveX Controls may cause problems for the document and the web page

(“Enable or Disable ActiveX”, 2012).

The specific web page features the participants were asked to use were the drop-down

list, check box, and text form options. Each feature utilizes a separate options menu allowing the

Usability Testing Project 9

user to customize the configuration of the web element. Our instructions ask our participants to

modify the drop-down list and check box features using their respective options menus. This is

not only done to enhance the web page, but to also allow us to assess their overall satisfaction

with the application’s menu system.

The instructions end with the participants viewing their web page in Mozilla Firefox and

comparing the layout of the web page within Mozilla Firefox and in Microsoft Word 2007. If the

instructions are completed as follows, the expected outcome is shown in figure 7 as it appears in

Web Layout view of Microsoft Word 2007. The same web page is shown in figure 8 as it

appears in Mozilla Firefox. When comparing the two views of the same document, it is clear that

there are some elements missing from the Mozilla Firefox representation of the document. We

asked our participants to note any differences they encountered during this step.

Figure 7: Web Page from Lab in Microsoft Word 2007

Usability Testing Project 10

Figure 8: Web Page from Lab in Mozilla Firefox

Logistics of the Usability Testing

Testing was conducted in the Towson University's Department of Computer Science

(COSC) Usability Testing Laboratory on the evening of May 7th. A group of 5 student-

colleagues from COSC 605 course were asked to perform a series of instructions designed to test

their ability with, and knowledge of, Microsoft Word 2007. The testing focused on each subject's

ability to successfully locate and invoke the Word 2007 Developer features necessary to

complete a series of steps designed to create a webpage and view the result using the IE and and

Firefox browsers; the overall test time was restricted to 9 minutes per subject.

The five subjects observed during testing were all males ranging in age from 22 to 29 with an

average age of 25.8 and a median age of 26. Two subjects described their industry as

academic/education and their position as student. The other three subjects listed Information

Technology as their industry and their position as either technical (2) or Other/engineer (1).

Four of the five test subjects claimed regular use of a Windows-based O/S. Of these four

subjects, one participant also listed regular use of the Mac O/S; one subject listed Mac and Linux

Usability Testing Project 11

as being used on a regular basis. The fifth test subject listed Mac as the sole O/S that was

regularly used.

When asked what versions MS Word they have used, two participants selected 2003,

2007 and 2010; one participant selected 2003 and 2007 while one listed 2003 and 2010. The fifth

listed MS Word 2011 for Mac.

All five participants claimed to be familiar with the Word text editor. One listed additional

familiarity with OpenOffice and Notepad; one acknowledged VIM (Linux) as an additional text

editor. Only the Mac-only user claimed to use Word on a daily basis.

Of the five categories (Novice, Beginner, Intermediate, Advanced and Expert) used to

rate overall familiarity with the MS Word ribbon, three subjects rated their overall familiarity as

Intermediate while two rated their familiarity with MS Word ribbon functionality as Advanced.

Each participant had previously created a web page using basic HTML.

Finally, it should be noted that participation in the testing was not voluntary, per se, as the

sample size was restricted to a subset of COSC 605 classmates. Each test "group" consisted of

the members from one of two other usability project groups in the class, each of which had 5

members. Note that the participant group which provided the subjects for this project was

randomly/arbitrarily assigned by the course instructor.

Metrics for Measuring Success

Considering the metrics used, two questionnaires were developed and completed by the

subjects. The first questionnaire was given to the subjects, right before the usability test to obtain

demographic information about the subjects, and assess their knowledge or familiarity with the

application.

Usability Testing Project 12

The test results differed from what we initially expected. The tasks were constructed

based on the metric called "completion rates", which is one of the usability fundamental metric

that provides specific result on the amount of time each subject took to complete the task. This

metric was chosen because it allows the researchers to easily measure subjects’ successful

completion of the task. This metric also helps assess the time spent on each specific subtask.

The first questionnaire purposed to gain insights into the subjects’ familiarity with MS

Word, and specifically MS Word 2007 used for the test.

The main metric we applied to the second questionnaire, also called the "After Effect",

aimed to evaluate the user's satisfaction level with the application, and the test as a whole. The

measure we used to estimate the subjects’ satisfaction is known as the post-task questionnaire or

rating with a single scale. This method looks very similar with the "completion rates" if the

"completion rates" metric is used with three or more implicatively-meaningful answers rather

than two ("yes" or "no"). This method was selected because it facilitates the interpretation and

reporting of results. The subjects were primarily asked to respond to the questions related to how

much they were satisfied with each specific task. In addition, they were supposed to respond to

few descriptive questions. The answers to those questions helped get the participants reactions

after the test. The subjects were for example asked to use one word to describe their experience,

to discuss what they found challenging during the test, and what could have made the activity

easier or enjoyable.

The inductive/deductive inference method will be used to extract information related to

our null hypothesis. The researchers assumed that "Advanced knowledge of Word

functionality/features has effect on the user's ability to create a webpage in a timely fashion",

from the questionnaires. The result of inferences will be reflected to usability problems.

Usability Testing Project 13

Results of the Testing

User Notes and Observations

1 Completed the task quickly, only asked for help when he couldn’t find the

developer’s toolbar, finished 1 minute early, very quick and sharp, moused

over icons in Word to figure out the name of the option

2 Had some trouble following some of the instructions, didn’t complete the

task, asked for a lot of assistance, didn’t quite follow all of the directions,

didn’t rely on hovering over icons to find out what they did, put dropdown

in the wrong place

3 had trouble figuring out check boxes, had some trouble overall, didn’t

finish, didn’t quite follow all of the directions, ignored step 5, messed up

step 13, used undo when he accidentally deleted everything, checked all

checkboxes

4 asked a lot of questions, completed the task, didn’t quite follow all of the

directions, didn’t skip step 4 and accomplished two tasks at once-- step 3

and 5, finished in 6 minutes with additional help

5 didn’t ask a lot of questions, completed the task, skipped a step and didn’t

go back to it, didn’t quite follow all of the directions, skipped step 4

Table 1: User results table

All of the users we tested have used Microsoft Word extensively at least weekly for over

7 years and the users didn’t think that the task was difficult at all, with 3 listing the task as

“reasonable” and 2 as “easy”. All of the users had made webpages in HTML before, and 2 users

ranked themselves as technical, 2 as students, and 1 as an engineer in terms of career positions.

However, only 3 of them completed the task, and only two followed all of the directions in

addition to completing the task.

Of the two who completed the task, one of them (User 4) had extensive Microsoft Word

experience (Word 2003, 2007, and 2010), yet asked a lot of questions, and was therefore guided

by the helper most of the time on the task. He marked the task as being easy and only had

difficulty locating the legacy tools. User 4 completed the task in 6 minutes. User 1, on the other

Usability Testing Project 14

hand, did not ask as many questions, and only relied on the helper for locating the developer’s

toolbar. He quickly completed everything else, however, and completed the task in 9 minutes,

with little to no guidance. User 1 marked the task as being reasonable, overall, and only has

experience with Word 2011 on a Mac. User 5 was the other user who finished the task, and he

managed to complete the task in a timely fashion. However, he did not follow all of the

directions, leaving out step 4, which is to add a page title to the web page. He did not rely on the

helper as much as User 4 did, and found the task easy.

Of the two users who didn’t complete the task, User 2 had the most trouble figuring out

the task, and asked the helper a lot of questions. He did not follow all of the directions on the

sheet, and ran out of time before getting to run the web page. He ranked the task as reasonable,

however. User 3 ranked the task as easy, and finding the developer’s toolbar as normal, however

seemed to struggle with the task. He accidentally deleted part of what he was doing, and it took

him a while to undo his changes because he didn’t realize it. He had the most trouble following

directions, as he ignored step 5, and messed up step 13. He didn’t rely on the helper as much, but

still had trouble finishing the task.

All of the users didn’t know where the Developer’s toolbar was in Word, so that’s where

most of the problems arose. The icons on the Developer’s toolbar don’t have any labels on them

to tell the user what they control. However, some (At least user 1 did.... possibly user 5) of the

users knew to hover over the icons to find out what controls which. The user who claimed to

have the most experience did not do that, and instead relied on the helper in the room with him to

give him guidance. None of the users relied on the help functionality in Word. If any of them

was stumped, they asked the helper for assistance. A couple of the users (At least 1) had some

Usability Testing Project 15

trouble using the dropdown list feature, because they didn’t know that they had to add all of the

drop-down list options at a time before closing out the popup box.

Wrap-up

After completion of their tasks, participants in the test were asked in the form of an

unstructured interview the following questions:

● How would you describe your overall experience?

● What challenge(s) did you experience when completing the tasks?

● Did you know MS Word 2007 had a web page design feature?

● Would you recommend the use of MS Word 2007 for webpage design?

● Why?

● Were you able to successfully complete your tasks?

● Were the tasks too many for the allotted time?

● Did you need help or more time to complete your task?

These questions were asked to gain insights into what users felt and thought about the tasks

and their overall experience. Inspired by Kirkpatrick’s (1976) evaluation model. The group used

level one of this model because it is not only recommended for instructional designers and

trainers, but also for multimedia developers. In addition, this model fits the purpose of this

experiment. The goal of the study was to measure the efficiency of MS Word 2007 webpage

feature. We assumed that users’ familiarity with MS Word 2007 had a positive impact on the

user's ability to complete the task in a timely fashion. Nevertheless, Kirkpatrick’s (1976) model,

also called “Happy sheets” or “Reaction sheets” helped measure the effectiveness and easiness to

use the webpage feature of MS Word 2007.

Usability Testing Project 16

Overall, participants enjoyed their experience. For all of them, the experience was fun.

However, out of five participants, only three completed the assigned task in less than nine

minutes. From these interviews we learned that the web page developer feature of MS Word

2007 was not a great tool for experienced web page developers/designers. In other words, the

more knowledgeable a participant was of programming language, the more likely he was to do

not complete the task or ask for help in completing his assigned task. In addition, familiarity with

MS Word 2007 did not result in participants’ successful completion of the task. However,

subjects who knew that MS Word was providing a web page design feature, tend to complete the

assigned task on lesser time than the other participants. These subjects also tend to recommend

this feature to novice; to individuals with no coding and programming background.

These interviews also revealed that the Icons, or the developers tools for example failed the

test of affordability, because the subjects had to hover over each icon to know what is was meant

for. Increasing the affordability of the web page development tools could cause the web page

developer in MS Word 2007 to be more efficient.

Usability Problems Noted.

Icons, text, developer tools, finding the developer tools:

[Problems]

1 From the questionnaires, we have noticed that most of the participants thought each task

has in-dependability between the tasks. Two of them said there is an overall process to

find the different type of form fields though. Thing is that every single participants didn't

know which button they were supposed to use for each task. In the other words, even

though each element of the "Legacy Tools" menu has been graphically visualized, they

Usability Testing Project 17

are not really distinguishable. The participants hover over icons in the "Legacy Tools"

menu repeatedly to find the appropriate one for each task.

2 All of the participants picked up the "Legacy Tools" menu as the most difficult part to

find even if they had have the instruction of it. It's because the menu looks like a button

rather than a menu because there is no difference between the menu and the buttons right

next to it. (In fact, it has little and rarely-visible south-directing arrow.) We were able to

know that it was the menu because its sub-menu with many legacy buttons were popped

up after clicking it.

3 The Developer menu tab stores all of the available web page functionalities for Microsoft

Word 2007; however, this menu is primarily used to perform more technical-based

actions such as writing macros. Since this does not correlate to the primary use of the

overall application, this menu is hidden by default, which makes it difficult for users to

understand where to locate HTML tools.

4 The ribbon interface was developed by Microsoft in an effort to assist users in finding the

most common used functions. While helpful, the submenus within the interface may

confuse some users when they attempt to follow instructions to locate specific features.

[Solutions]

1 Each icon seems to be small and similar with each other. Thereby, it is difficult to find

each icon directly. In this case, if we add the name of each element next to each

visualized icon respectively, we can mitigate their ambiguity.

2 As removing similarity between the "Legacy Tools" menu button and the other buttons,

we can solve this problem. So, we recommend making it as a section named "Legacy

Tools" instead of the button-like designed original menu icon.

Usability Testing Project 18

3 To accommodate the issue of the elusive Developer menu tab, Microsoft could modify

their future applications to include a separate menu for web page options that is not

hidden from view. These tools should be offered outside of the Developer tab because

certain options, such as the Legacy Forms tools, can be used without the use of Microsoft

Visual Studio.

4 As seen in figures 2-5, the submenus within each menu tab have their submenu heading

located at the bottom of each section. Since the rest of the application adopts a top-down

design in terms of its interface, it may benefit the application to relocate the submenu

headers to the top of each section rather than the bottom. This suggested change is

proposed in figure 9 below with an example created using the Home menu.

Figure 9: Suggested Change to Submenu Headers

Proposed Changes and Priorities

It could be argued that of the four solutions proposed in the previous Section, a subjective

ranking order in terms of importance could be 3, 2, 4, 1. In terms of perceived engineering time,

the possible order from least to most time could be 4, 1, 2, 3.

The most important thing exposed by the testing is the fact that the Developer menu tab

simply isn't visible, or easily accessible apparently experienced users. The idea of turning the

Developer menu "on" by default is a relatively easy fix to startup setting. The hard engineering

would be to develop a separate menu for web page options and offering them outside of the

Usability Testing Project 19

Developer tab; that's development work and a code project with some impact on Word at the

more integrated Microsoft Office level. In terms of importance, it is of the first rank; in terms of

engineering tasking it would probably be the most engineering intensive. Therefore, unless the

marketing indicates that highlighting the Developer menu as a standalone feature would result in

a significant positive impact on the Office suite's bottom line, it falls down the priority list in the

engineering pecking order of projects that need to/can be done in a future major release of

Windows.

Of the other proposed changes, the fourth (moving the submenu heading to the top of

each section) would seem to be not only a logical suggestion but a workable project making it a

desirable target for an engineering “quick fix.” The immediate engineering impact would be,

essentially, inverting logic that already exists in the code. That's the kind of project often

relegated to new-hires or even motivated summer interns. It's both doable, narrowly focused and

shouldn't require a massive engineering investment. Again, unless such a modification could be

proven to have a minimal investment (dollar) impact in an engineering budget for a major

release, it would have to bear the scrutiny of a marketing analysis.

Solutions 1 and 2 become somewhat of a toss-up. Solution 1 is perhaps the least

important in terms of impact (although it solves a problem of ambiguity which is always a

desirable goal) but it would probably take the least amount of time to code. Solution 2, on the

other hand, seems to involve a redesign that may have more moving parts that can be easily

accommodated or foreseen. However, in terms of positive impact on the Word Development

environment, Solution 2 is perhaps the second most important proposal.

Unfortunately, all user community suggestions of this type end up in a bucket of

proposed “nice to haves” that every engineering organization from Facebook to Google to

Usability Testing Project 20

Microsoft to IBM to Apple draws upon for possible inclusion in a future release. A limited, but

lengthy, sample size of engineering experience has lent itself to the observation that within any

engineering community there is a noticeable distinction between development and maintenance.

The culture of new versus old that expresses itself in the identification of, and the potential care

for, legacy code.

Development engineering, regardless of company or culture, tends to throw things “over

the wall” and walk away. This has implications - both obvious as well as subtle - for changes

proposed to existing product lines. If any portion of the Word ribbon is deemed to be legacy

code, it much harder, if not impossible, to institute change/modification because the original

coders have long since moved onto their next assignment s and the engineering resources

assigned to support the components of released products would be considered maintenance

coders whose sole responsibility is to, in effect, cash-cow the application, or feature, and not to

invest in further, future development.

What did you learn?

As a group we understood that we could not have control over all the conditions

surrounding the test. For example, we did not expect some participants to be anxious, or nervous

few minutes before taking the test. As a group we predicted that individuals’ familiarity with

MS word, would complete the task in less than 10 minutes. We assumed that the successful

candidate was to be the one that completes all the tasks in 6 minutes, and comes up with a

webpage closely identical to the sample provided to each of the participants. However, our null

hypothesis was not validated by our results. Participants’ familiarity with MS Word did not

result in the subjects meeting our performance level.

Usability Testing Project 21

The low performance of some of the subjects, despite their familiarity with MS Word

could be explained by participants’ anxiety, or nervosity. It is also our belief that giving

participants instructions before they entered the lab could have reduced the effects of anxiety and

nervosity on our subjects’ performance. The difference in performances among our subjects

could also be explained by the fact that the subjects who were the first to be tested came back,

and informed the remaining participants on the nature of tasks. This may explain why participant

four completed in six minutes, and participant five did not ask lots of questions; while the first

participant took nine minutes to complete the tasks.

During the group testing post-mortem, we also found that our test environment and

delivery could be improved. As an example, we had to carefully reconstruct the before/after data

in order to correctly match the identities of the test subjects. While we were able to successfully

reconstruct (match) each subject questionnaire with the appropriate debrief results, neglecting to

initially number subject information for tracking caused a loss of valuable time.

It was the group consensus of that we provided too much information to each subject

during the testing. This included showing each subject how to invoke the Developer features in

the ribbon - a point about which we debated in terms of design right up to the moment of testing.

Initially, we did not include time to reset the environment (between test subjects 1 and 2) to the

default state in which the Developer features were hidden. Technically, this would have

invalidated the second subject's results.

The note taker neglected to keep copies of the instructions in order to mark each subject's

progress. This caused the note take to make notes in a slower, periodic fashion which could have

resulted, and more than likely did result, in the loss of valuable observation time thereby

distracting the official observer who had to tell the note taker what was needed to be captured.

Usability Testing Project 22

The observer and note taker roles could have been improved by using additional copies of the

user instruction sheets as templates for taking notes.

It was speculated that if the group had been able to perform a trial run on itself (as a first

set of subjects), the results of such a trial run may have exposed some of the flaws described

above thereby improving the overall delivery of the test process as well as serving to mitigate a

number of flaws the actual testing exposed.

Therefore, the results of our usability test cannot be generalized. Nevertheless, we believe that

following our recommendations, the developers at Microsoft could improve the usability scores

of the web page feature in MS Word 2007, in terms of easiness to learn, efficiency and

effectiveness. We did compare MS Word 2007 with MS Word 2003, and realized that in MS

Word 2003, the webpage feature was easy to learn, and had great affordability. This came as a

surprise to us because Word 2007 is technically more advanced than Word 2003. It could have

been a great adventure to run a usability test on this feature in MS Word 2003, Word 2007, and

Word 2010. However, as group, comparing MS 2003 with 2007 led us to the conclusion that in

one application it is impossible to meet all the usability goals.

Usability Testing Project 23

References

Armstrong, William B. Planning Your Survey: A Brief Primer on Strategies and Approaches.

University of California, San Diego.

http://studentresearch.ucsd.edu/sriweb/Presentations/PlanningYourSurvey.pdf

Honebean, P. (n.d). Retrieved May 16, 2012 from Indiana University Bloomington. Wiki:

http://en.wikiversity.org/wiki/Level_One_Survey_Example

Kirkpatrick, D.L. (1976). Evaluation of training. In Craig, R.L. (Ed.), Training and

Development Handbook (pp.18-27). New York: McGraw-Hill.

http://en.wikipedia.org/wiki/Microsoft_Office_2007 (retrieved 18may2012).

http://psyc.queensu.ca/subpool/debriefingform.doc

http://social.msdn.microsoft.com/search/en-us?query=create+webpage+%2B+Word+2007

(retrieved 17May2012).

Microsoft (2012). Enabled or Disable ActiveX Settings in Office Documents. From

http://office.microsoft.com/en-us/word-help/enable-or-disable-activex-controls-in-office-

documents-HA010031067.aspx?CTT=1

Microsoft (2012). Getting Started with Microsoft Office 2007. From

http://office.microsoft.com/en-us/support/getting-started-with-microsoft-office-2007-

FX101839657.aspx

Microsoft (2012). Insert a Drop-down List, Check Box, or Other Items in a Form. From

http://office.microsoft.com/en-us/word-help/insert-a-drop-down-list-check-box-or-other-

items-in-a-form-HP005189943.aspx?CTT=1

Usability Testing Project 24

Microsoft (2012). Show the Developer Tab or Run in Developer Mode. From

http://office.microsoft.com/en-us/word-help/show-the-developer-tab-or-run-in-developer-

mode-HA010173052.aspx?CTT=1

Microsoft (2012). Word 2007 Training Courses. From http://office.microsoft.com/en-us/word-

help/word-2007-training-courses-HA010215566.aspx