Aimee Maree Forsstrom

Supervisor Dr Graham Cooper

Honours Presentation Southern Cross University

Web Site Navigation Design with Respect to Cognitive Load

• Interdisciplinary research– Computer Science– Educational Psychology

• Cognitive Load and Human Computer Interaction– Does web site navigation design effect how we

take in information– How can we apply Cognitive Load Theory to

web site navigation design to assist the intake of information

The story so far…

Timeline - Key Dates

• Honours Thesis started in February 2008

• Original plan to run cognitive load tests on Australian Learning and Teaching Council June 2008

• First Honours presentation July 2008

• Honours update presentation October 2008

• Plan changed to be run on a mock up University Faculty website January 2009

• Plan changed to be run on a online game presenting random facts to the user May 2009

• Ethics received in August 2009

• Experiment ran first two weeks of September 2009

• Final Honours presentation October 2009

Research in the Social Network era• Participants informed online

– Tweets placed on www.twitter.com– Message sent to friends on www.facebook.com– Link posted on www.facebook.com profile page– Email sent to all Southern Cross University staff/students– Email sent to all Programming Society members (UTS)

• 202 participants played the game

• 124 meet needed requirements

• 118 participants in the final clean data set – 59 in control group– 59 in treatment group

Participants in both groups balanced on age/gender/comp use

What we learnt

• Social networking is a great way to reach large number of participants

• Participants numbers coming to the site will peak and then slowly die off

• Online game was a great tool to drive people through website navigation

• People don’t mind playing a game and answering some short questions

• Expect a percentage of drop out’s and possible technical glitches (40% error rate encountered)

• Track mouse paths through all stages of the site to ensure proper human activity (not a robot)

• Data must be thoroughly cleaned and results checked to ensure there are no anomalies from technical issues or cheating

Experiment Design

Playing a Game can be fun

• Online fact learning game design chosen

• 15 questions presented in random fashion

• Answers where found by navigating through the web site menu

• 5 top levels of categories in menu

• 4 lower level menu options per category

• Using a game we can drive the participant through the menu – maximum use of menu 15 clicks with short time

frame 10min

• Engage participant in web site material and appeal to wide audience

Experiment Components

• Research online check

• Random fact game play – 10min, 15 questions

• Effort survey

• Fill in facts, participant to fill in missing words from 10 facts presented in game

• Menu rebuild, participant asked to rebuild the menu to show if they acquired the menu schema

• Usability Survey

1. Random Game start page presented

2. Participant asked to scroll to bottom of page and roll dice

3. After the dice is rolled participant is given a question and asked to locate the section in the menu that will hold the answer

4. The menu sections are named after key words in the question asked

5. If the participant clicks on the wrong menu they are asked to re-read the question and try again

6. Once the final answer is read the participant is directed to answer questions about the game

Experiment Steps

Data Analysis Methods

Statistics

• Non Parametric and parametric statistics used

• Distribution was heavily skewed so Parametric statistics could not be used on most questions

• Qualitative and Quantitative data analysed

Survey asked on Effort

• Two questions

• Questions answered by 7 point likert scale

• Mann Whitney U Test

• Descriptive Statistics

Survey asked on Usability

• Two questions

• Questions answered by 7 point likert scale

• Mann Whitney U Test

• Descriptive Statistics

• Three open Ended questions

• Linear regression on positive and negative word count

• Linear regression and on word categories

Fill in Facts Questions

• 10 Fill in fact questions grouped by easy, medium, hard

• Mann Whitney U test on overall score average

• ANOVA – Question type score and group interaction– Variables Easy, Medium, Hard questions, Factor group id

• Chi-Square test for each individual question – Group comparison of the number of participants who got

the question right

• Wilcoxon Ranked Sum Test – Difference in individual question answers between groups

• Group Average for Fill in facts question completion time

Menu Schema Rebuild

• Menu Rebuild, 18 menu answers captured

• Mann Whitney U test on overall score average

• ANOVA – Menu top, lower menu rebuild and group interaction– Variables top, lower level menu sections, Factor group

id

• Chi-Square test for each individual menu element – Group comparison of the number of participants who

got the question right

• Wilcoxon Ranked Sum Test – difference in individual menu section answers between

groups

• Group Average for Menu rebuild time

• Mouse path collected for each group (screen click point captured)– Comparison of mouse click path used in game

• Overall random facts game time– T-test of game time means

• Random Facts navigation time between each of the 15 questions– T-test of individual navigation steps

Mouse Paths and Game Time

Preliminary Results

Effort Survey Likert Question One• Effort one question displays slight backward

trend– How easy was it to use the menu to play the

game

Mann-Whitney Test

Ranks

  groupN Mean Rank Sum of Ranks

effort_one_e

 

experiment 59 63.48 3745.50

treatment 59 55.52 3275.50

Total 118    

Test Statisticsa

effort_one_eMann-Whitney U 1505.500

Wilcoxon W 3275.500

Z -1.344

Asymp. Sig. (2-tailed) .179

a. Grouping Variable: group

Treatment group mean 2.05 Control group mean 1.85

1 = very easy2 = easy

• Statistical Significance found for Effort question two– How distracting did you find the scrolling

Effort Survey Likert Question Two

Mann-Whitney Test

Ranks

  group N Mean Rank Sum of Rankseffort_two

 

treatment 59 51.23 3022.50

control 59 67.77 3998.50

Total 118    

Test Statisticsa

effort_two_eMann-Whitney U 1252.500

Wilcoxon W 3022.500

Z -2.687

Asymp. Sig. (2-tailed) .007a. Grouping Variable: group

Treatment group mean 2.71 Control group mean 3.51

2 = very slightly distracting3 = slightly distracting4 = distracting

Usability Survey Likert Questions• No difference in likert scale questions

– After rolling the dice for each question locating the navigation menu was easy

• Treatment group mean = 5.37• Control group mean = 5.34

– I enjoyed playing the random facts game• Treatment group mean = 4.97• Control group mean = 4.97

4 = neither agree nor disagree5 = slightly agree6 = agree

Usability Survey Open Ended

• Qualitative data was analyzed for positive and negative words mentioned

– Experiment Group had higher positive score– Treatment Group had higher negative score

– 45% of the treatment group asked to change the scrolling in the game

– 12% of the experiment group asked to change the scrolling in the game

Fill in Facts Questions

• Statistical significance between groups for overall question score– Treatment group outperformed the control

group on overall test scoresMann-Whitney Test

Ranks

  groupN Mean Rank Sum of Ranks

overall_score_c

 

treatment group 59 68.02 4013.00

control group 59 50.98 3008.00

Total 118    

Test Statisticsa

overall_score_c

Mann-Whitney U 1238.000

Wilcoxon W 3008.000

Z -2.726

Asymp. Sig. (2-tailed) .006a. Grouping Variable: group

Treatment group mean 13.56Control group mean 12.05

Maximum Score for facts = 17

Fill in Facts Questions

• Statistical difference found for overall mark on question type and group id– Treatment group outperformed the control

group on the medium to harder questions

ANOVA

Sum of Squares df Mean Square F Sig.

easy_score_c Between Groups .008 1 .008 .009 .923

Within Groups 105.932 116 .913    Total 105.941 117      

med_score_c Between Groups 7.627 1 7.627 3.713 .056

Within Groups 238.271 116 2.054    Total 245.898 117      

hard_score_c Between Groups 22.915 1 22.915 10.015 .002

Within Groups 265.424 116 2.288    Total 288.339 117      

Fill in Facts Questions

• Statistical difference found for individual question marks between group– Treatment group outperformed control group

question before_score after_score sign_of_difference positive_rank negative_rank1 46 41 -   92 48 57+ 14  3 53 49 -   54 49 54+ 9  5 54 56+ 1.5  6 43 39 -   57 47 52+ 9  8 54 56+ 1.5  9 31 56+ 17  

10 50 45 -   911 53 57+ 5  12 6 22+ 16  13 16 29+ 15  14 39 40+ 1  15 44 50+ 12  16 42 50+ 13  17 42 47+ 9  

123 T = 28

Menu Schema Rebuild

• Statistical significance not found on overall scores– Difference in means 14.03 for treatment group

and 12.02 for control group not significant

Treatment group mean 14.03 Control group mean 12.02

Maximum Score for facts = 33

Mann-Whitney Test

Ranks

  groupN Mean Rank Sum of Ranks

menu_overall_score

 

treatment 59 64.41 3800.00

control 59 54.59 3221.00

Total 118    

Test Statisticsa

menu_overall_score

Mann-Whitney U 1451.000

Wilcoxon W 3221.000

Z -1.560

Asymp. Sig. (2-tailed) .119a. Grouping Variable: group

• Statistical significance found on number of correct answers between groups– Experiment group outperformed control group

Menu Schema Rebuild

question control  group experiment group sign_of_difference positive_rank negative_rank4 55 54 -   25 36 35 -   2

16 16 15 -   210 20 22+ 4  6 30 27 -   6

18 12 9 -   615 10 13+ 6  2 47 51+ 8.5  

17 16 20+ 8.5  7 22 17 -   10.5

14 13 18+ 10.5  3 43 49+ 12  8 26 19 -   13.51 35 42+ 13.5  9 32 40+ 15.5  

12 19 27+ 15.5  19 4 13+ 17  13 19 29+ 18  11 9 22+ 19  

148 T = 42

Lessons Learnt

• Need twice as many questions – 30 instead of 15

• keep participants engaged longer – game play average was 5min not 10min

• More elements in menu schema rebuild – 30 instead of 18

• Need for more medium and heavy questions– 10/15 instead of 6/10 easy/hard question ratio

• Ensure wording of questions crosses borders

• Include more survey questions

• Perform more technical checks and pilot

Thank you