Project Pass 2:

Mid-High Fidelity Prototyping

& User Evaluation

Team Trottoir Stephan Bouthot

Erin Bush

Jihyung Im

Pui Yan (Denise) Kwok

Chin­Tsai (Amy) Tsai

Table of Contents

Section A: Prototype and Redesign Rationale A1. Redesign A3. Prototype Illustrations

Section B: Planning User Evaluations B1. Evaluation Protocol

Our Experiment Our Procedure

Section C: User Evaluations C1. Subjects C2. Evaluation Results C3. Design Rationale C4. Design Process Reflection

Section D: Resource Management Management of Resources Appendices

Appendix A: Evaluation Tools Appendix B: Evaluation Raw Data Appendix C: Consent Forms

Section A: Prototype and Redesign Rationale

A1. Redesign

The high­level plan for our next prototype is to remove the hierarchy of our system.

From the users surveyed and the design review, we learned that a meeting host among

university students is generally an arbitrary designation. The goal of this next design is,

therefore, to remove the need for a host to create and approve meeting times.

The need for event creation cannot be avoided, but the user to create it will no longer

have any privileges or duties greater than the users he/she invites to the meeting. The

meeting time will be chosen algorithmically, much like the suggested meeting time which in

previous iterations the host could ignore. All users with the link will be able to upload their

schedule and will be messaged when a meeting time is determined.

A few issues involving error prevention in our previous prototype were brought up as

well. Specifically, the ability to edit or undo past actions by the user as their schedule changes

or as mistakes are noticed. There is no reason to prevent this, as users’ calendars are

ostensibly independent but users with the link to the event will be able to change their

schedule. A malicious user may take advantage of this to force a meeting to occur at a

specific time, but this cannot be helped.

The previous prototype received many complaints about the all­or­nothing way it treats

schedules. As such, we will be adding a level for availability between the current “would like to

meet at this time” and the “cannot meet at this time”. This specification will not only give us a

way to break ties, all else being equal, but will give users a more accurate way to represent

their schedule. A user presented with only two options might select undesirable times as

unavailable in the hopes of manipulating the selected meeting time in their favour. The third

option should, at the cost of a bit more complexity, give the user a more accurate calendrical

representation of their schedules and the feeling of a less arbitrary outcome.

Surveyed users expressed uncertainty in the progress of inputting their schedules. To

fix this, we have improved the visibility of the system state by adding a progress bar at the top

partitioned into steps. At any point during interaction with the website, the user will be able to

see what has been done, what is presently being done, and what is left to do. A confusion

also arose from importing calendars and whether the imported data will be useful. We see the

necessity of the import calendar function from our studies that indicated a vast majority of

students (over 70%) use an online calendar and they have to go back and forth between the

calendar and the scheduling tool. Thus, the second prototype will allow users to import

calendars and indicate in the user’s availability screen (where previously they could only

indicate their availability) when they are busy, this will reduce the human errors caused by the

back and forth while still givingthe freedom of selecting availability as per meeting/event to the

users.

The second prototype iteration focused on developing a horizontal prototype. The time

scheduling solution does not contain a lot of functionality, it mostly focuses on user

experience and ease of use, so a horizontal prototype will give more useful user evaluation

feedback.

A3. Prototype Illustrations

Section B: Planning User Evaluations

B1. Evaluation Protocol

The focus of our evaluations is answering “How much time does an attendee take to

input their availability on Junction relative to the amount of time taken to input their availability

on Doodle.” We have chosen this because it is the most common and time consuming task

carried out in the meeting scheduling process; an event creation will take one person to

create the event but every invitee will need to input their availability.

Although there are many scheduling solutions available, we will be comparing the time

of our software to only Doodle. This way, our evaluation scope is narrowed to an acceptable

experiment time per experimentee. Doodle is one of the most popular scheduling tool among

students, with a slogan “Find a date for a meeting 2x faster” so we feel that if we are proven

faster than their scheduler, we have a competitive advantage.

Our Experiment

The summary of our experiment is as follows: Null H0: There is no difference in the amount of time to input availability between Junction and Doodle

Alternate H1: Using the Junction interface results in faster availability input Independent variable: Which interface is used

Dependent variable: Time taken to complete task Nuisance variables: Previous experience with Doodle

Our Procedure

With these parameters in mind, the following are the steps taken to complete the experiment.

1. User signs consent form

2. User input their availability using Junction & Doodle, the experimenter will time the

duration of each response and validate that the user enters the correct information

a. Junction with Schedule A with Calendar Import

b. Doodle with Schedule B

c. Junction with Schedule A

d. Junction with Schedule B with Calendar Import

e. Doodle with Schedule A

f. Junction with Schedule B

3. User answers questionnaire

a. What method of scheduling do you use the most often?

b. Have you used Doodle before?

i. How often do you use Doodle for a large group (10+ people)?

ii. How often do you use Doodle for a small group (<10 people)?

iii. How often do you use Doodle with 20+ options?

iv. How often do you use Doodle with <20 options?

c. Do you use an electronic calendar? (Google Calendar, etc.)

Variability in the experiment is reduced by alternating Junction and Doodle, having the

experimentee schedule two times using two separate schedule, and alternating using

Schedule A and Schedule B in case the user remembers the schedule. The scenario for

Schedule A is scheduling a meeting for a large group and the availability cannot be predicted

so Schedule A has many free times and the complementing Doodle will have a lot of choices.

On the contrary, Schedule B is for a smaller group working on a group project so the user will

want to meet in between classes and end early enough to commute home so Schedule B

does not have a lot of free time and the complementing Doodle will not have a lot of choices.

Because of the way Doodle is designed, we predict our results to show that Schedule B is

faster on Doodle whereas Schedule A is faster on Junction. From our previous market

research, Doodle is more commonly used for Schedule A scenario but we will again verify this

through the questionnaire. Junction is used twice for each schedule because we feel that the

import feature greatly affects the time, but not all students use electronic calendars so our

experiment data will be more complete with times using the import function and without. Refer

to appendix for the two schedules, screenshots of the two Doodle polls used with the

schedules, all the consent forms used in the experiment, and what the users see after using

the import function the calendar.

A functional web app prototype using HTML, CSS, and JavaScript with no back end

will be used in the evaluations. Users will be able to see all the screens and input all the

necessary data. The import of calendars will be simulated. Two separate web pages are used

for the two different schedules so when the user click the import button, the correct

corresponding calendar will be “imported” by filling in the available time slots with unavailable

blocks. However, the users will not be able to interact with the invitation portion of the design

because there are multiple media where the user can send an invite and outside of what we

can influence.

Section C: User Evaluations

C1. Subjects

Our goal in this prototype was to target university students. In our experience as

university students, there has not been a collaborative meeting scheduler that meets our

needs. Things that set university student meetings apart from the norm are odd meeting

hours, scattered and conflicting schedules, and strange sleep schedules. Whereas with usual

meetings, there is a general trend of free time occurring in predictable blocks between work

ending in the afternoon and sleep starting at night, student schedules do not have as much of

a pattern. Our goal in this prototype was to meet the (in our opinion, overlooked) needs of

students.

User 1 Graduating university student

Executive within an undergraduate

society that regularly schedules large

(20+ people) meetings

~20 people must be present,

other optional attendees are

also encouraged to attend

Schedules group project meetings

User 2 Graduating university student

Juggles multiple group projects and

commitments

Has frequent group meetings, and

frequently had to respond to Doodle

and Facebook poll, but doesn’t have

much experience initiating them

himself.

User 3 Graduating university student

Plans a minimum of 3 meetings a

week

Have multiple group projects

Needs to have multiple one on one

interviews with candidates for the

tutoring events he plans

User 4 A 3rd year university student

Trying to find the balance between

an eventful social life and school

Volunteers at many student events

User 5 Graduating university student

Plans 200+ people events with 5+

other people

Has too many commitments that he

sometimes forget about

commitments he made

User 6 Graduating University student, head

of a student society

Has the power to choose what

software everyone in her student

society will use

Sets up more than 5 meetings a

week with other busy students

Has to invite 5 to 30 people to

various meetings

User 7

Graduating university student, previously head of a student society

Had to use Doodle and Outlook frequently in the past, but has not found a good

scheduling tool.

Has a busy schedule with group projects and other commitments, and would like to

save time scheduling meetings.

Our experiment subjects were all university students in their twenties with experience

scheduling meetings. This was to get an accurate idea of how our UI performed with our

target demographic. Some of these users were part of the pass 1 survey. The pass 1 survey

was an open­ended questionnaire and the prototype has changed so much from pass 1 that

there should be little to no issues that would bias the data.

C2. Evaluation Results

Results (in minute:second)

Schedule A Schedule B

Junction ­ with calendar import

Junction ­ no calendar import

Doodle Junction ­ with calendar import

Junction ­ no calendar import

Doodle

User 1 5:57 4:51 4:13 0:45 1:42 1:05

User 2 4:21 1:47 2:27 0:38 0:28 0:33

User 3 4:20 1:57 2:03 0:56 0:18 1:18

User 4 4:38 4:00 1:52 0:43 0:30 0:55

User 5 4:10 3:23 2:39 0:59 1:04 1:09

User 6 3:50 2:58 4:20 0:40 0:30 0:45

User 7 4:10 1:20 3:50 0:35 0:50 0:40

Average 4:29 2:53 3:03 0:45 0:46 0:55

Comparing the reported time durations for the users to complete the tasks under

different scenarios, we found that given schedule A, which had a more wider range of

availabilities, users were significantly faster in filling out their availabilities using Doodle and

Junction without the import calendar function than using Junction with the calendar import

feature. On average, it took Doodle 32% less time to complete. In contrast, for schedule B

with a narrower and simpler availability ranges, Junction performed better ­ on average by

18%. Interestingly, users completed filling out their availability significantly faster without using

the calendar import feature with Schedule A, and negligibly so with schedule B. However, we

do observe that this may have been due to the overhead of users learning the system, using

Junction with Schedule A and the calendar import was the first trial we ran.

A potential problem with the evaluation process that we observed is that there was a

definite learning curve when first using Junction, which may have added extra time to

complete the task. Also, the drag and drop functionality on our prototype was sometimes

buggy which would have brought on extra challenge to the users. Although our subject

contained a good representative user group, 8 subjects is still a small sample size, and our

result could have been improved with a bigger sample size. Another problem was how

realistic the imported calendar is. The prototype had a few slots with busy times but

realistically, students would have due dates for tasks, more events, and other uses of a

calendar; this was not well validated. The purpose of the import calendar feature was to

improve the time to determine availability which is not the focus of this experiment and is very

difficult to simulate.

Questionnaire Results: All users have used Doodle

4 out of 7 students say they use Doodle or a combination of Doodle & Facebook the

most out of other scheduling methods

Remaining 3 use Facebook or Any.do the most

All users use either Google Calendar (the most), Facebook, or Mac Calendar

Use of Doodle

Doodle for 10+

people

Doodle for <10

people

Doodle for 20+

time slots

Doodle for <20

time slots

User 1 80% 20% 30% 70%

User 2 0% 100% 90% 10%

User 3 50% 50% 20% 80%

User 4 0% 100% 50% 50%

User 5 50% 50% 100% 0%

User 6 60% 40% 70% 30%

User 7 0% 100% 20% 80%

Average 34% 66% 54% 46%

From the results of the questionnaire, university students either use Facebook, by

asking for available times in a chat, or use Doodle. Doodle is designed for small amounts of

time to choose from but from our questionnaire, we see that students definitely use Doodle

when there are a lot of time slots and when they invite a large group of people. Students still

choose to use chats to set up meeting times which proves that they are unsatisfied with

Doodle and they do not know of other schedulers that is more suited to their needs. Again, the

questionnaire results are tainted with the limited sample size problem, the results would

definitely be more valid with more students from different schools and faculties. 100% of the

users use an electronic calendar which indicates more effort can be put in the “Import from

Calendar” feature.

A possible bias that could be avoided with a larger pool of possible subjects is

familiarity with Doodle’s interface. All the subjects had used Doodle prior to the experiment,

and that facility may have cause them to improve their times. Two possible solutions to this

problem present themselves: a) find students who have not used Doodle before, a daunting

task; or b) give our subjects a period of time in which to familiarize themselves with Junction’s

interface. The former may be difficult, but the latter comes with problems of its own.

Convincing subjects to spend twenty minutes inputting fake calendars was tricky enough, but

add another twenty minutes and annoyance may set in, biasing the study in a different way.

C3. Design Rationale

The click­and­drag functionality did increase user satisfaction as the user feedback we

got that it was “more visual” and looked more like their online calendars. Many users tried to

drag on the Doodle interface when they were inputting a lot of times. This part of the design

worked well, but our prototype needs to be fixed to eliminate the bugs and reduce the time it

takes for users to learn how to use our system. A dire bug is the inconsistent behaviour

between browsers.

Some improvements we could potentially add are the following:

The default box size is two hours, research can be done to determine the most

suitable default time slot

User currently need to click then drag to resize the time slot, the user should be able to

drag without the click

Allow user to resize by dragging the top, currently its dragging the bottom only

Allow user to create time slots that cross multiple days

Our interface can improve by reducing the amount of work we require our users to do.

Our interface requires users to input their entire schedule, when users have a busy and

dispersed schedule (Schedule A), Doodle was easier to use, as they just needed to verify

yes/no for a small set of times. We can improve this by also reducing the amount of times a

user needs to drag by showing only the parts of the calendar the host identified as available,

and allowing users to click­and­drag within those times instead of within the entire week. We

can also improve this by saving the times that users drag in one event to their account and

allowing them to share the same available times to a second event invite by clicking just one

button.

The current state of our prototype was developed after taking into consideration the

feedback we received from the heuristic evaluation and design reviews for our paper

prototype. We have improved the major issue of missing features (like event name and

description editing) as well as the increasing the freedom the user has by allowing users to

easily go from one step to the previous step or to the next step, providing more visual cues to

indicate that the user can undo his/her previous action.

Our experiment does not provide a clear indication of an advantage over Doodle but

many potential biases, leaning towards Doodle, have been identified. We cannot say if

Junction is better or not, we need more experiments and data to be sure. An interesting

discovery through the questionnaires is the frequent use of Facebook. Once we determine

how the students use Facebook to schedule meetings, we could either incorporate that

method into Junction or convert Junction into a Facebook app if the students use it because

everyone is on the same platform.

C4. Design Process Reflection

Our initial prototype was a horizontal high­fidelity prototype, as we wanted to design a

new collaborative meeting scheduling experience. Instead of having a host specify only the

times that he/she wants to meet, we also allow participants to specify their availability, even if

it means the host did not specify that he/she was available. We changed our prototype slightly

to be more vertical in the “specify availability” feature, as this was the feature that we believed

our prototype could improve, and designed a user evaluation to verify this.

While Doodle was undeniably the faster interface in this study, there were elements of

our design that users expressed preference for. The dragging feature in particular was

mentioned by users as a nice way to enter information. If we were to continue working on this

design, the next step would be to incorporate parts of Doodle that users enjoyed as well. One

of the largest problems with this prototype was the requirement for the user to work around

sleep. A nice feature of Doodle is that it only provides participants with certain times for them

to choose between. The middle ground may be the best route in this case, giving the host a

way to remove times that will certainly not work for a majority of participants.

Going through the process of designing and validating definitely improved our insight

into designing. 12 weeks ago, the faults in our design were difficult to see but with the help of

professors and our peers, we became better at using the system from the perspective of a

user and see how we think our interface works differs from the mental model that users build

of our interface. Designing the user evaluation made us think deeply about how we should

design it so that the result is not tainted with bias and variables. Looking back, the first draft of

the experiment was not well thought­out. The experiment that we carried out is still not

perfect, it could have been more compact and possibly include testing other features as well if

time allows. The biggest flaw was the prototype as pointed out before, if we gave it more

thought, we would have a more definitive result.

We also discovered that doing frequent informal evaluations throughout the design

process can help us quickly identify what users want, how they think and how to improve our

design. We performed informal evaluations and heuristic evaluations as a team throughout

the design process, that allowed us to get feedback regarding confusing language or features

and observe when and where users get stuck within the interface. This helped us a lot in our

design process as we were able to quickly change our paper prototype and improve before

preparing a medium fidelity prototype and performing formal user evaluations. Hopefully in the

future we will require fewer external assistance and be able to see obvious flaws ourselves.

Section D: Resource Management

Management of Resources

Chin­Tsai (Amy) Tsai Cost estimation is based on hourly wage of $40 for contract consulting work in Canada

Completed Tasks Time taken Cost

Designing Hi­fidelity Prototype 4 hours $160

Designing Experiment 2 hours $80

Developing Prototype 1 hour $40

Conducting Evaluations 2 hours $80

Writing Pass 2 Report 8 hours $320

Erin Bush Cost estimation is based on hourly wage of $40 for contract consulting work in Canada

Completed Tasks Time taken Cost

Designing Hi­Fidelity Prototype 4 hours $160

Designing Experiment 3 hours $120

Developing Prototype 10 hours $400

Stephan Bouthot Cost estimation is based on hourly wage of $40 for contract consulting work in Canada

Completed Tasks Time taken Cost

Writing pass 2 8 hours $320

Designing experiment 2 hours $80

Design reviews 1 hour $40

Testing prototype 2 hours $80

Video 1 hour $40

Pui Yan Kwok Cost estimation is based on hourly wage of $40 for contract consulting work in Canada

Completed Tasks Time taken Cost

Designing iterative Paper Prototypes 4 hours $160

Designing High­Fidelity Prototype 5 hours $200

Designing experiment 2 hours $80

Conducting Evaluations 2 hours $80

Writing Pass 2 Report 8 hours $640

Jihyung Im Cost estimation is based on hourly wage of $40 for contract consulting work in Canada

Completed Tasks Time taken Cost

Conducting user experiments 3 hours $120

Analysing and evaluating experiment results 2 hours $80

Writing Pass 2 Report 1 hours $40

Designing lo­fi prototype 2 hours $80

Refining list of requirements 2 hours $80

Heuristic evaluation 1 hour $40

Appendices

Appendix A: Evaluation Tools

Schedule A:

Time Sunday

Apr 12

Monday

Apr 13

Tuesday

Apr 14

Wed.

Apr 15

Thursday

Apr 16

Friday

Apr 17

Saturday

Apr 18

0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:00

11:00

12:00

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

Schedule B:

Time Sunday

Apr 12

Monday

Apr 13

Tuesday

Apr 14

Wed.

Apr 15

Thursday

Apr 16

Friday

Apr 17

Saturday

Apr 18

0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:00

11:00

12:00

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

Doodle for Schedule A:

Doodle for Schedule B:

Imported Times:

Appendix B: Evaluation Raw Data

Name of Participa

nt

Signed form?

Trial 1

Trial 2

Trial 3

Trial 4

Trial 5

Trial 6

Most Used

Scheduler

Used Doodle?

Doodle for 10+ people

Doodle for <10

people

Doodle for 20+ options

Doodle for <20 options

electronic calendar?

Nicholas Zeng Yes 5:57 1:05 4:51

0:45

4:13 1:42

doodle + fb msgs yes

80% of time

20% of time

30% of time

70% of time

Google calendar

Sae Young Kim Yes 4:21 0:33 1:47

0:38

2:27 0:28

Facebook poll

Yes, but never

initiates it himself Never

Not very

often. 2­3

times Almost always

Almost never

Google calendar

Derek Lun Yes 4:20 1:18 1:57 0:56

2:03 0:18 FaceBook Yes

Yes, 50%

Yes, 50%

1/month

1/4 months

Google Calendar

Sammi Jiang Yes 4:38 0:55 4:00

0:43

1:52 0:30 AnyDo Yes Never

1/week 50% 50% Mac

Vineet Mahendru Yes 4:10 1:09 3:23

0:59

2:39 1:04

FaceBook/Doodle Yes 3/year 3/year 100% 0

Google/FaceBook

Diana Kim Yes 3:50 0:45 2:58 0:40

4:20 0:30 Doodle Yes

3 times a

month

two times

a month 70% 30%

Google Calendar

Sean Lee Yes 4:10 0:40 1:20 0:35

3:50 0:50

Doodle/Outlook Yes Never Yes Rarely

Almost always

Google Calendar

Average 4:29 0:55 2:53

0:45

3:03 0:46

Appendix C: Consent Forms