Project Pass 2:
Mid-High Fidelity Prototyping
& User Evaluation
Team Trottoir Stephan Bouthot
Erin Bush
Jihyung Im
Pui Yan (Denise) Kwok
ChinTsai (Amy) Tsai
Table of Contents
Section A: Prototype and Redesign Rationale A1. Redesign A3. Prototype Illustrations
Section B: Planning User Evaluations B1. Evaluation Protocol
Our Experiment Our Procedure
Section C: User Evaluations C1. Subjects C2. Evaluation Results C3. Design Rationale C4. Design Process Reflection
Section D: Resource Management Management of Resources Appendices
Appendix A: Evaluation Tools Appendix B: Evaluation Raw Data Appendix C: Consent Forms
Section A: Prototype and Redesign Rationale
A1. Redesign
The highlevel plan for our next prototype is to remove the hierarchy of our system.
From the users surveyed and the design review, we learned that a meeting host among
university students is generally an arbitrary designation. The goal of this next design is,
therefore, to remove the need for a host to create and approve meeting times.
The need for event creation cannot be avoided, but the user to create it will no longer
have any privileges or duties greater than the users he/she invites to the meeting. The
meeting time will be chosen algorithmically, much like the suggested meeting time which in
previous iterations the host could ignore. All users with the link will be able to upload their
schedule and will be messaged when a meeting time is determined.
A few issues involving error prevention in our previous prototype were brought up as
well. Specifically, the ability to edit or undo past actions by the user as their schedule changes
or as mistakes are noticed. There is no reason to prevent this, as users’ calendars are
ostensibly independent but users with the link to the event will be able to change their
schedule. A malicious user may take advantage of this to force a meeting to occur at a
specific time, but this cannot be helped.
The previous prototype received many complaints about the allornothing way it treats
schedules. As such, we will be adding a level for availability between the current “would like to
meet at this time” and the “cannot meet at this time”. This specification will not only give us a
way to break ties, all else being equal, but will give users a more accurate way to represent
their schedule. A user presented with only two options might select undesirable times as
unavailable in the hopes of manipulating the selected meeting time in their favour. The third
option should, at the cost of a bit more complexity, give the user a more accurate calendrical
representation of their schedules and the feeling of a less arbitrary outcome.
Surveyed users expressed uncertainty in the progress of inputting their schedules. To
fix this, we have improved the visibility of the system state by adding a progress bar at the top
partitioned into steps. At any point during interaction with the website, the user will be able to
see what has been done, what is presently being done, and what is left to do. A confusion
also arose from importing calendars and whether the imported data will be useful. We see the
necessity of the import calendar function from our studies that indicated a vast majority of
students (over 70%) use an online calendar and they have to go back and forth between the
calendar and the scheduling tool. Thus, the second prototype will allow users to import
calendars and indicate in the user’s availability screen (where previously they could only
indicate their availability) when they are busy, this will reduce the human errors caused by the
back and forth while still givingthe freedom of selecting availability as per meeting/event to the
users.
The second prototype iteration focused on developing a horizontal prototype. The time
scheduling solution does not contain a lot of functionality, it mostly focuses on user
experience and ease of use, so a horizontal prototype will give more useful user evaluation
feedback.
A3. Prototype Illustrations
Section B: Planning User Evaluations
B1. Evaluation Protocol
The focus of our evaluations is answering “How much time does an attendee take to
input their availability on Junction relative to the amount of time taken to input their availability
on Doodle.” We have chosen this because it is the most common and time consuming task
carried out in the meeting scheduling process; an event creation will take one person to
create the event but every invitee will need to input their availability.
Although there are many scheduling solutions available, we will be comparing the time
of our software to only Doodle. This way, our evaluation scope is narrowed to an acceptable
experiment time per experimentee. Doodle is one of the most popular scheduling tool among
students, with a slogan “Find a date for a meeting 2x faster” so we feel that if we are proven
faster than their scheduler, we have a competitive advantage.
Our Experiment
The summary of our experiment is as follows: Null H0: There is no difference in the amount of time to input availability between Junction and Doodle
Alternate H1: Using the Junction interface results in faster availability input Independent variable: Which interface is used
Dependent variable: Time taken to complete task Nuisance variables: Previous experience with Doodle
Our Procedure
With these parameters in mind, the following are the steps taken to complete the experiment.
1. User signs consent form
2. User input their availability using Junction & Doodle, the experimenter will time the
duration of each response and validate that the user enters the correct information
a. Junction with Schedule A with Calendar Import
b. Doodle with Schedule B
c. Junction with Schedule A
d. Junction with Schedule B with Calendar Import
e. Doodle with Schedule A
f. Junction with Schedule B
3. User answers questionnaire
a. What method of scheduling do you use the most often?
b. Have you used Doodle before?
i. How often do you use Doodle for a large group (10+ people)?
ii. How often do you use Doodle for a small group (<10 people)?
iii. How often do you use Doodle with 20+ options?
iv. How often do you use Doodle with <20 options?
c. Do you use an electronic calendar? (Google Calendar, etc.)
Variability in the experiment is reduced by alternating Junction and Doodle, having the
experimentee schedule two times using two separate schedule, and alternating using
Schedule A and Schedule B in case the user remembers the schedule. The scenario for
Schedule A is scheduling a meeting for a large group and the availability cannot be predicted
so Schedule A has many free times and the complementing Doodle will have a lot of choices.
On the contrary, Schedule B is for a smaller group working on a group project so the user will
want to meet in between classes and end early enough to commute home so Schedule B
does not have a lot of free time and the complementing Doodle will not have a lot of choices.
Because of the way Doodle is designed, we predict our results to show that Schedule B is
faster on Doodle whereas Schedule A is faster on Junction. From our previous market
research, Doodle is more commonly used for Schedule A scenario but we will again verify this
through the questionnaire. Junction is used twice for each schedule because we feel that the
import feature greatly affects the time, but not all students use electronic calendars so our
experiment data will be more complete with times using the import function and without. Refer
to appendix for the two schedules, screenshots of the two Doodle polls used with the
schedules, all the consent forms used in the experiment, and what the users see after using
the import function the calendar.
A functional web app prototype using HTML, CSS, and JavaScript with no back end
will be used in the evaluations. Users will be able to see all the screens and input all the
necessary data. The import of calendars will be simulated. Two separate web pages are used
for the two different schedules so when the user click the import button, the correct
corresponding calendar will be “imported” by filling in the available time slots with unavailable
blocks. However, the users will not be able to interact with the invitation portion of the design
because there are multiple media where the user can send an invite and outside of what we
can influence.
Section C: User Evaluations
C1. Subjects
Our goal in this prototype was to target university students. In our experience as
university students, there has not been a collaborative meeting scheduler that meets our
needs. Things that set university student meetings apart from the norm are odd meeting
hours, scattered and conflicting schedules, and strange sleep schedules. Whereas with usual
meetings, there is a general trend of free time occurring in predictable blocks between work
ending in the afternoon and sleep starting at night, student schedules do not have as much of
a pattern. Our goal in this prototype was to meet the (in our opinion, overlooked) needs of
students.
User 1 Graduating university student
Executive within an undergraduate
society that regularly schedules large
(20+ people) meetings
~20 people must be present,
other optional attendees are
also encouraged to attend
Schedules group project meetings
User 2 Graduating university student
Juggles multiple group projects and
commitments
Has frequent group meetings, and
frequently had to respond to Doodle
and Facebook poll, but doesn’t have
much experience initiating them
himself.
User 3 Graduating university student
Plans a minimum of 3 meetings a
week
Have multiple group projects
Needs to have multiple one on one
interviews with candidates for the
tutoring events he plans
User 4 A 3rd year university student
Trying to find the balance between
an eventful social life and school
Volunteers at many student events
User 5 Graduating university student
Plans 200+ people events with 5+
other people
Has too many commitments that he
sometimes forget about
commitments he made
User 6 Graduating University student, head
of a student society
Has the power to choose what
software everyone in her student
society will use
Sets up more than 5 meetings a
week with other busy students
Has to invite 5 to 30 people to
various meetings
User 7
Graduating university student, previously head of a student society
Had to use Doodle and Outlook frequently in the past, but has not found a good
scheduling tool.
Has a busy schedule with group projects and other commitments, and would like to
save time scheduling meetings.
Our experiment subjects were all university students in their twenties with experience
scheduling meetings. This was to get an accurate idea of how our UI performed with our
target demographic. Some of these users were part of the pass 1 survey. The pass 1 survey
was an openended questionnaire and the prototype has changed so much from pass 1 that
there should be little to no issues that would bias the data.
C2. Evaluation Results
Results (in minute:second)
Schedule A Schedule B
Junction with calendar import
Junction no calendar import
Doodle Junction with calendar import
Junction no calendar import
Doodle
User 1 5:57 4:51 4:13 0:45 1:42 1:05
User 2 4:21 1:47 2:27 0:38 0:28 0:33
User 3 4:20 1:57 2:03 0:56 0:18 1:18
User 4 4:38 4:00 1:52 0:43 0:30 0:55
User 5 4:10 3:23 2:39 0:59 1:04 1:09
User 6 3:50 2:58 4:20 0:40 0:30 0:45
User 7 4:10 1:20 3:50 0:35 0:50 0:40
Average 4:29 2:53 3:03 0:45 0:46 0:55
Comparing the reported time durations for the users to complete the tasks under
different scenarios, we found that given schedule A, which had a more wider range of
availabilities, users were significantly faster in filling out their availabilities using Doodle and
Junction without the import calendar function than using Junction with the calendar import
feature. On average, it took Doodle 32% less time to complete. In contrast, for schedule B
with a narrower and simpler availability ranges, Junction performed better on average by
18%. Interestingly, users completed filling out their availability significantly faster without using
the calendar import feature with Schedule A, and negligibly so with schedule B. However, we
do observe that this may have been due to the overhead of users learning the system, using
Junction with Schedule A and the calendar import was the first trial we ran.
A potential problem with the evaluation process that we observed is that there was a
definite learning curve when first using Junction, which may have added extra time to
complete the task. Also, the drag and drop functionality on our prototype was sometimes
buggy which would have brought on extra challenge to the users. Although our subject
contained a good representative user group, 8 subjects is still a small sample size, and our
result could have been improved with a bigger sample size. Another problem was how
realistic the imported calendar is. The prototype had a few slots with busy times but
realistically, students would have due dates for tasks, more events, and other uses of a
calendar; this was not well validated. The purpose of the import calendar feature was to
improve the time to determine availability which is not the focus of this experiment and is very
difficult to simulate.
Questionnaire Results: All users have used Doodle
4 out of 7 students say they use Doodle or a combination of Doodle & Facebook the
most out of other scheduling methods
Remaining 3 use Facebook or Any.do the most
All users use either Google Calendar (the most), Facebook, or Mac Calendar
Use of Doodle
Doodle for 10+
people
Doodle for <10
people
Doodle for 20+
time slots
Doodle for <20
time slots
User 1 80% 20% 30% 70%
User 2 0% 100% 90% 10%
User 3 50% 50% 20% 80%
User 4 0% 100% 50% 50%
User 5 50% 50% 100% 0%
User 6 60% 40% 70% 30%
User 7 0% 100% 20% 80%
Average 34% 66% 54% 46%
From the results of the questionnaire, university students either use Facebook, by
asking for available times in a chat, or use Doodle. Doodle is designed for small amounts of
time to choose from but from our questionnaire, we see that students definitely use Doodle
when there are a lot of time slots and when they invite a large group of people. Students still
choose to use chats to set up meeting times which proves that they are unsatisfied with
Doodle and they do not know of other schedulers that is more suited to their needs. Again, the
questionnaire results are tainted with the limited sample size problem, the results would
definitely be more valid with more students from different schools and faculties. 100% of the
users use an electronic calendar which indicates more effort can be put in the “Import from
Calendar” feature.
A possible bias that could be avoided with a larger pool of possible subjects is
familiarity with Doodle’s interface. All the subjects had used Doodle prior to the experiment,
and that facility may have cause them to improve their times. Two possible solutions to this
problem present themselves: a) find students who have not used Doodle before, a daunting
task; or b) give our subjects a period of time in which to familiarize themselves with Junction’s
interface. The former may be difficult, but the latter comes with problems of its own.
Convincing subjects to spend twenty minutes inputting fake calendars was tricky enough, but
add another twenty minutes and annoyance may set in, biasing the study in a different way.
C3. Design Rationale
The clickanddrag functionality did increase user satisfaction as the user feedback we
got that it was “more visual” and looked more like their online calendars. Many users tried to
drag on the Doodle interface when they were inputting a lot of times. This part of the design
worked well, but our prototype needs to be fixed to eliminate the bugs and reduce the time it
takes for users to learn how to use our system. A dire bug is the inconsistent behaviour
between browsers.
Some improvements we could potentially add are the following:
The default box size is two hours, research can be done to determine the most
suitable default time slot
User currently need to click then drag to resize the time slot, the user should be able to
drag without the click
Allow user to resize by dragging the top, currently its dragging the bottom only
Allow user to create time slots that cross multiple days
Our interface can improve by reducing the amount of work we require our users to do.
Our interface requires users to input their entire schedule, when users have a busy and
dispersed schedule (Schedule A), Doodle was easier to use, as they just needed to verify
yes/no for a small set of times. We can improve this by also reducing the amount of times a
user needs to drag by showing only the parts of the calendar the host identified as available,
and allowing users to clickanddrag within those times instead of within the entire week. We
can also improve this by saving the times that users drag in one event to their account and
allowing them to share the same available times to a second event invite by clicking just one
button.
The current state of our prototype was developed after taking into consideration the
feedback we received from the heuristic evaluation and design reviews for our paper
prototype. We have improved the major issue of missing features (like event name and
description editing) as well as the increasing the freedom the user has by allowing users to
easily go from one step to the previous step or to the next step, providing more visual cues to
indicate that the user can undo his/her previous action.
Our experiment does not provide a clear indication of an advantage over Doodle but
many potential biases, leaning towards Doodle, have been identified. We cannot say if
Junction is better or not, we need more experiments and data to be sure. An interesting
discovery through the questionnaires is the frequent use of Facebook. Once we determine
how the students use Facebook to schedule meetings, we could either incorporate that
method into Junction or convert Junction into a Facebook app if the students use it because
everyone is on the same platform.
C4. Design Process Reflection
Our initial prototype was a horizontal highfidelity prototype, as we wanted to design a
new collaborative meeting scheduling experience. Instead of having a host specify only the
times that he/she wants to meet, we also allow participants to specify their availability, even if
it means the host did not specify that he/she was available. We changed our prototype slightly
to be more vertical in the “specify availability” feature, as this was the feature that we believed
our prototype could improve, and designed a user evaluation to verify this.
While Doodle was undeniably the faster interface in this study, there were elements of
our design that users expressed preference for. The dragging feature in particular was
mentioned by users as a nice way to enter information. If we were to continue working on this
design, the next step would be to incorporate parts of Doodle that users enjoyed as well. One
of the largest problems with this prototype was the requirement for the user to work around
sleep. A nice feature of Doodle is that it only provides participants with certain times for them
to choose between. The middle ground may be the best route in this case, giving the host a
way to remove times that will certainly not work for a majority of participants.
Going through the process of designing and validating definitely improved our insight
into designing. 12 weeks ago, the faults in our design were difficult to see but with the help of
professors and our peers, we became better at using the system from the perspective of a
user and see how we think our interface works differs from the mental model that users build
of our interface. Designing the user evaluation made us think deeply about how we should
design it so that the result is not tainted with bias and variables. Looking back, the first draft of
the experiment was not well thoughtout. The experiment that we carried out is still not
perfect, it could have been more compact and possibly include testing other features as well if
time allows. The biggest flaw was the prototype as pointed out before, if we gave it more
thought, we would have a more definitive result.
We also discovered that doing frequent informal evaluations throughout the design
process can help us quickly identify what users want, how they think and how to improve our
design. We performed informal evaluations and heuristic evaluations as a team throughout
the design process, that allowed us to get feedback regarding confusing language or features
and observe when and where users get stuck within the interface. This helped us a lot in our
design process as we were able to quickly change our paper prototype and improve before
preparing a medium fidelity prototype and performing formal user evaluations. Hopefully in the
future we will require fewer external assistance and be able to see obvious flaws ourselves.
Section D: Resource Management
Management of Resources
ChinTsai (Amy) Tsai Cost estimation is based on hourly wage of $40 for contract consulting work in Canada
Completed Tasks Time taken Cost
Designing Hifidelity Prototype 4 hours $160
Designing Experiment 2 hours $80
Developing Prototype 1 hour $40
Conducting Evaluations 2 hours $80
Writing Pass 2 Report 8 hours $320
Erin Bush Cost estimation is based on hourly wage of $40 for contract consulting work in Canada
Completed Tasks Time taken Cost
Designing HiFidelity Prototype 4 hours $160
Designing Experiment 3 hours $120
Developing Prototype 10 hours $400
Stephan Bouthot Cost estimation is based on hourly wage of $40 for contract consulting work in Canada
Completed Tasks Time taken Cost
Writing pass 2 8 hours $320
Designing experiment 2 hours $80
Design reviews 1 hour $40
Testing prototype 2 hours $80
Video 1 hour $40
Pui Yan Kwok Cost estimation is based on hourly wage of $40 for contract consulting work in Canada
Completed Tasks Time taken Cost
Designing iterative Paper Prototypes 4 hours $160
Designing HighFidelity Prototype 5 hours $200
Designing experiment 2 hours $80
Conducting Evaluations 2 hours $80
Writing Pass 2 Report 8 hours $640
Jihyung Im Cost estimation is based on hourly wage of $40 for contract consulting work in Canada
Completed Tasks Time taken Cost
Conducting user experiments 3 hours $120
Analysing and evaluating experiment results 2 hours $80
Writing Pass 2 Report 1 hours $40
Designing lofi prototype 2 hours $80
Refining list of requirements 2 hours $80
Heuristic evaluation 1 hour $40
Appendices
Appendix A: Evaluation Tools
Schedule A:
Time Sunday
Apr 12
Monday
Apr 13
Tuesday
Apr 14
Wed.
Apr 15
Thursday
Apr 16
Friday
Apr 17
Saturday
Apr 18
0:00
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:00
9:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
Schedule B:
Time Sunday
Apr 12
Monday
Apr 13
Tuesday
Apr 14
Wed.
Apr 15
Thursday
Apr 16
Friday
Apr 17
Saturday
Apr 18
0:00
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:00
9:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
Doodle for Schedule A:
Doodle for Schedule B:
Imported Times:
Appendix B: Evaluation Raw Data
Name of Participa
nt
Signed form?
Trial 1
Trial 2
Trial 3
Trial 4
Trial 5
Trial 6
Most Used
Scheduler
Used Doodle?
Doodle for 10+ people
Doodle for <10
people
Doodle for 20+ options
Doodle for <20 options
electronic calendar?
Nicholas Zeng Yes 5:57 1:05 4:51
0:45
4:13 1:42
doodle + fb msgs yes
80% of time
20% of time
30% of time
70% of time
Google calendar
Sae Young Kim Yes 4:21 0:33 1:47
0:38
2:27 0:28
Facebook poll
Yes, but never
initiates it himself Never
Not very
often. 23
times Almost always
Almost never
Google calendar
Derek Lun Yes 4:20 1:18 1:57 0:56
2:03 0:18 FaceBook Yes
Yes, 50%
Yes, 50%
1/month
1/4 months
Google Calendar
Sammi Jiang Yes 4:38 0:55 4:00
0:43
1:52 0:30 AnyDo Yes Never
1/week 50% 50% Mac
Vineet Mahendru Yes 4:10 1:09 3:23
0:59
2:39 1:04
FaceBook/Doodle Yes 3/year 3/year 100% 0
Google/FaceBook
Diana Kim Yes 3:50 0:45 2:58 0:40
4:20 0:30 Doodle Yes
3 times a
month
two times
a month 70% 30%
Google Calendar
Sean Lee Yes 4:10 0:40 1:20 0:35
3:50 0:50
Doodle/Outlook Yes Never Yes Rarely
Almost always
Google Calendar
Average 4:29 0:55 2:53
0:45
3:03 0:46
Appendix C: Consent Forms
Top Related