D4.8.3 BLUE Experiment Results and Evaluation v1.0

7/27/2019 D4.8.3 BLUE Experiment Results and Evaluation v1.0

1/39

This deliverable presents the results of the BLUE experiment, regarding the

EXPERIMEDIA components employed by the "My Museum Story" and "My Museum

Guide" applications and regarding the usability, efficiency and effectiveness of the

applications. The document outlines the experiment process, presents and analyses the results

and discusses the experiment outcomes.

D4.8.3

BLUE Experiment Results and Evaluation

2013-10-07

George Lepouras, Angeliki Antoniou, Jenny Rompa, Costas Vassilakis

(University of Peloponnese)

Ioanna Lykourentzou, Yannick Naudet, Eric Tobias (Henri Tudor Public

Research Centre)

www.experimedia.eu


2/39

EXPERIMEDIA Dissemination level: PU

Copyright University of Peloponneseand other members of the EXPERIMEDIA consortium 2013 2

Project acronym EXPERIMEDIA

Full title Experiments in live social and networked media experiences

Grant agreement number 287966

Funding scheme Large-scale Integrating Project (IP)

Work programme topic Objective ICT-2011.1.6 Future Internet Research andExperimentation (FIRE)

Project start date 2011-10-01

Project duration 36 months

Activity 4 Experimentation

Workpackage 4.8 EX8: BLUE

Deliverable lead organisation University of Peloponnese

Authors George Lepouras, Angeliki Antoniou, Jenny Rompa, CostasVassilakis (University of Peloponnese)

Ioanna Lykourentzou, Yannick Naudet, Eric Tobias (Henri

Tudor Public Research Centre)

Reviewers Sergiusz Zieliski (Pozna Supercomputing and NetworkingCenter)

Stephen C. Phillips (IT Innovation Centre)

Version 1.0

Status Final

Dissemination level PU: Public

Due date PM24 (2013-09-30)

Delivery date 2013-10-07


3/39



Table of Contents

1. Executive summary ............................................................................................................................ 5

2. Introduction ........................................................................................................................................ 6

3. Experiment overview ......................................................................................................................... 7

3.1. Experiment design .................................................................................................................... 7

3.1.1. Changes since D4.8.2 ........................................................................................................... 7

3.1.2. Target group .......................................................................................................................... 7

3.1.3. Experimental factors ............................................................................................................ 7

4. Experiment results ............................................................................................................................ 11

4.1. Method...................................................................................................................................... 11

4.1.1. Qualitative analysis .............................................................................................................. 12

4.1.2. Quantitative evaluation metrics ........................................................................................ 12

4.2. Analysis of experimental results ............................................................................................ 13

4.2.1. Facebook game results analysis ......................................................................................... 13

4.2.2. On-site results analysis ....................................................................................................... 18

4.2.3. Usefulness of cognitive style in recommendations ........................................................ 22

5. EXPERIMEDIA component assessment .................................................................................... 24

5.1. EXPERIMEDIA Experiment Content Component (ECC) ............................................ 24

5.2. EXPERIMEDIA Social Content Component (SCC) ....................................................... 24

6. Discussion .......................................................................................................................................... 25

7. Conclusion ......................................................................................................................................... 29

8. References .......................................................................................................................................... 30

Appendix A. Semi-structured interview ............................................................................................. 31

Appendix B. Avatars, pets and tools Icons ....................................................................................... 37

B.1. Avatar icons ............................................................................................................................. 37B.2. Pet icons ................................................................................................................................... 37

B.3. Tool icons ................................................................................................................................. 38

Appendix C. Museum templates ......................................................................................................... 39


4/39



List of Tables

Table 1. Phase T0 - Factors ....................................................................................................................... 8

Table 2. Factors captured by the questionnaire ...................................................................................... 8

Table 3. Users' Estimated and Actual Cognitive Style ........................................................................ 14

Table 4: opics associated to exhibitions, corresponding to cognitive style dimensions. ............. 23


5/39



1.Executive summaryThe goal of the BLUE experiment of EXPERIMEDIA is to explore the use of visitors' cognitive

styles and content interest in order to personalize their experiences inside a museum. The related

experimental runs were conducted at the Foundation of the Hellenic World (FHW) in Athens.The goals and requirements for the experiment have been described in EXPERIMEDIA

deliverable D4.8.1 and the initial experiment design in D4.8.2.

Four experimental sessions were conducted on four different dates. The total number of

participants was 30: 15 male and 15 female with an average age of 30. Of the 30 participants 6

people visited alone, 17 with one or more friends, and 7 with family members. The sampling

processes followed replicated the processes the museum will apply in order to attract new

visitors, advertise exhibitions and also collect relevant information for user profiling.

Experiments revealed that a good proportion of visitors liked and followed recommendations,had a positive impression of using the recommender and reported the mobile guide enhanced

their quality of experience. The Facebook game could predict user interests and cognitive profile.

Furthermore, visitors were happy with the provided Point of Interest descriptions. Overall the

results are promising, as the analysis of our experimental runs has shown the usefulness of the

BLUE technologies and their suitability to improve user experience.


6/39



2. IntroductionFollowing their design and implementation, the EXPERIMEDIA BLUE applications, "My

Museum Story" (MMS) and "My Museum Guide" (MMG) were assessed in an experiment

performed at the Hellenic Cosmos Cultural Centre of the Foundation of Hellenic World inAthens, Greece.

The experiment's aim was twofold: first, to investigate the extent to which the EXPERIMEDIA

BLUE experiment achieved its goal of enhancing the Quality of Experience for visitors within

the museum, and second, to assess in a real-world setting the EXPERIMEDIA components

used in building the BLUE applications.

The rest of the document is structured as follows: section 3 gives a brief overview of the

experiment, focusing on the changes that had to take place since the initial description of the

experiment in D4.8.2, the target group, recorded variables and experiment structure. Section 4presents the data collection process, and the methodology employed during data analysis as well

as the experiments' results. Section 5 describes the assessment of the two EXPERIMEDIA

components used in the development of EXPERIMEDIA BLUE, namely the EXPERIMEDIA

Experiment Content Component (ECC) and the EXPERIMEDIA Social Content Component

(SCC). Section 6analyzesthedata collected during the experiment in relation to the hypotheses

made and discusses the findings. The document ends with final conclusions and future work in

section 7. The template of the questionnaire employed during experiments can be found at

Appendix A.


7/39



3.Experiment overview3.1. Experiment design3.1.1.Changes since D4.8.2Since the time Deliverable 4.8.2 (describing the initial experiment design) was submitted, changes

in FHW exhibitions and ticketing policy caused alterations on the experiment's design. The

Mathematical exhibition was replaced by a new one, titled "Your Planet Needs You!", which

offers glimpses into the world of 2050 and explores how mankind will survive on a changing

planet. Due to this exhibition change, new content had to be created for the recommendation

engine to cater for different user profiles. That resulted in abandoning the geolocalised message

posting functionality, in an effort to make up for the extra time and effort, as well as to keep up

with the original trial schedule. Furthermore, this new exhibition had a more classical monolithic

structure, where the visitor has to follow a specific path, making it impossible to build additional

virtual exhibitions like we expected to do (see D4.8.2, section 2.2.2). The small number of items

to recommend (8 in total plus cafeteria and gift shop), and the fact that 6 of them were shows,

which have a fixed schedule and were played most often once per day, meant that we could not

give visitors the choice between multiple paths or visit sequences. Actions have been proposed

to visitors by the MyMuseumGuide (MMG)application on the sole basis of their interaction with

the system, since it was found that the location system was not sufficiently reliable.

To help attract subjects for the experiments, FHW altered its ticketing policy offering free

entrance for the visitors invited to our trials. We could therefore have a better mix of subjects,

with different backgrounds as opposed to what had been originally envisaged in D4.8.2 (section2.2.3) for phase 2, i.e. the controlled experiment. To this end, Phase 2 and 3 described in D4.8.2

(section 2.2.3) were merged in one single phase of open experiment where for each run we had 7

or 8 participants. As anticipated in D4.8.2 for the open experiment, not all visitors had played

the MyMuseumStory (MMS) Facebook game.

3.1.2.Target groupChanges in ticketing policy helped us attract more subjects for our experiments. The experiment

was advertised through Facebook. The majority of the subjects were Facebook users who played

the MMS game, while the rest had learnt of the experiment from friends. Nevertheless, the

number of participants was small.

3.1.3.Experimental factorsAs defined in Deliverable 4.8.2, there were 4 main experimental phases: T0, before the user goes

to the museum; T1 and T2 which, for the purpose of variable generation can be treated as one

step, the preparation of the visit and visit of the museum; and T3, after the visit has been

concluded.

At T0, different factors were captured, all related to the user. Of these factors, some provided

demographic information and others were related to the users game choices and behaviour on

the social network. Table 1 summarises the identified factors:


8/39



Table 1. Phase T0 - Factors

Category Factors

Demographics Age

Gender

Mother Tongue

Facebookbehaviour

Facebook ID as provided by Facebook,

Number of Facebook friends

Game choices Avatar Choices: Old-Wise, Artist, Engineer, Scientist, Alien, Rapper, TV-Persona,Mad Scientist, or Diplomat

Pet Choices: Cat, Dog, Gold Fish, Monkey, or Owl

Tool Choices: Book, Clock, Disco Ball, or Heart

Choice of Museum Template

Number and type of exhibits they have won playing the gameNumber and type of exhibits they have placed in-game

Number of games they have played

Also, during phase T0 and after the above factors had been collected, a number of other factors

were computed. From the players game choices, their cognitive styles were estimated, since

different choices were related to different dimensions of the cognitive style. Similarly, peoples

interests were also computed both from their game choices and their cognitive profiles.

Additional factors, collected during the experiment in T1|2, had been elaborated and committedin a Google document. These encompass the visitor's: identifier, position, visited exhibitions,

pictures taken, comments on pictures and choice to upload their visit to Facebook.

A visitor's identifier is either their Facebook identifier, if they chose to provide it by logging in,

or a randomly generated UUID. It is used to keep track of a visitor's use of the system. A

visitor's position is provided as a geographical position in latitude and longitude as separate but

related variables. A visitor's position is used to establish their visiting style. The visitor's

engagement with the system is measured by the number of comments and the number of

pictures taken using the devices. Their readiness to use and interactivity with social media is

measured by their willingness to upload their visit, sharing it with their social network.

After the museum visit, in T3, a questionnaire was used to gather data from the visitors. This

data had been organized into different categories. A template of the questionnaire is included as

appendix. Table 2 summarises the questionnaire categories and questions.

Table 2. Factors captured by the questionnaire

Category Factors

General Participant

Information

A visitor's Facebook ID (if known) to relate the questionnaire to data

collected in T0, T1 and T2.Participants Name


9/39



Category Factors

Educational Background

Whether they were part of a group and if so, whether they held the tablet.

Whether they played the Facebook game

The short version of the MBTI cognitive style questionnaire

My Museum Story relatedquestions

Avatar choice, pet choice, tool choice

Difficulty in making these choices

Rationale behind these particular choices

Museum layout template choice

Rationale behind this particular choice

Self-reported visiting style

Number of exhibits they collected

Problem regarding process of choosing exhibits

Rationale for choosing specific exhibits

Whether they invited friends to the game

Whether they published their game results on Facebook

Whether they enjoyed playing the game

Likes and dislikes regarding the game's features

Improvements suggestions

Their stance on games for cultural institutions

My Museum Guide relatedquestions

Exhibition rating (interesting or not)

Whether or not the exhibition in question was recommended by the

application

Whether they mainly followed the recommendation

Whether they would upload photos and a diary of their visit to Facebook

Rating ease of use (of MMG app)

Whether they enjoyed using the application

Whether the presentation of the information was satisfying

Additional information they would have liked to see

Whether they would make changes in the presentation of therecommendations

Whether they would have preferred seeing a mapWith whom they visited

In case they visited with a group:

1. Whether they followed the group's decisions or the app'srecommendations

2. Whether they checked other group member's screens

In case they visited alone, whether they followed the app'srecommendations

Their opinion on advantages and disadvantages of the mobile application

Preferences while visiting a museum

Minimum and maximum time they would have liked to spend at the FHW


10/39



Category Factors

Whether MMS and/or MMG helped make their visit more enjoyable.


11/39



4.Experiment resultsAccording to the success scale proposed in D4.8.1 section 8, the BLUE experiment has achieved

a moderate success and some steps towards success have been realized. The criterion for

baseline success, i.e. being an EXPERIMEDIA test bed, has been fulfilled since the ECC hasbeen fully integrated and used, and knowledge and expertise gathered accordingly. Then, the

analysis of our experimental runs has shown the usefulness of the BLUE technologies and their

suitability to improve user experience. QoE has been evaluated qualitatively, thus the impact of

using BLUE technologies has been shown. Four research hypotheses have been evaluated based

on both qualitative and statistical analysis.. QoS has been taken into account at design time.

4.1. MethodFour experimental sessions were conducted. The first took place on 18 th June 2013 with 8

participants, the second on 23rd June 2013 with 7 participants, the third on 30th June 2013 with 8

participants and the fourth on 16th July with 7 participants. The total number of participants was

30, 15 male and 15 female with an average age of

30. Of the 30 participants 6 people visited alone,

17 with one or more friends, and 7 with family

members (there were also two children visiting

with their parents but they were not included in

the sample due to legal constraints). Consistent

with the literature describing how people usually

visit museums in a group (Antoniou, 2009), our

sample also reflected this trend. In addition, datacollected from participating families is of

significant value as data regarding family museum

visit behaviour is still lacking.

Summarizing, the experimental procedure required invited visitors to first play the Facebook

game, then visit the museum and use our mobile application and finally fill in a questionnaire.

More specifically:

Two applications were developed, MyMuseumStory

(MMS) and MyMuseumGuide (MMG). MMS is a

Facebook game giving its users the ability to

populate their own virtual museum with various

exhibits. At first, users are asked to choose an

avatar, a pet and a tool to play with. Based on their

choices, the MMS calculates and extracts each users

cognitive style. Moreover, the chosen virtual

museums template indicates the usersvisiting style.

As the user wanders in her museum she can win

objects to add as exhibits by playing mini-games. All

exhibits earned and placed inside the virtualmuseum throughout the game are recorded and stored as they reveal users personal interests.

Figure 1. Visitor playing MMS game

Figure 2. Visitor using MMG application


12/39



The MMG computes recommendations in the form of predictions. These are then provided in

the form of a sequence of Points of Interest (POI) to the visitor. The procedure of computing

recommendations has been published in (Naudet, 2013). After his visit, the visitor is asked to fill

a questionnaire, giving his personal likings and opinions.

The experiments provided data that can be used to evaluate the experiment both qualitatively

and quantitatively.

4.1.1.Qualitative analysisThe qualitative analysis examines the effect of multiple factors in regards to the Facebook game,

the effect that the on-site experiment had on visitors' QoE and, as an additional element, briefly

discusses QoS (Quality of Service) handling during the on-site experiments.

Facebook game (MMS):

Gameplay choices Game usability

QoE during on-site experiment. We examine the effect of the following factors:

Overall Visitor Satisfaction and Evaluation Photo Option and Online Visit Sharing (MMG) Tablet Usability (MMG) Effect of Personalization and Content Adaptation (MMG) Group Visitors Evaluations

Additional element:

QoS evaluation4.1.2.Quantitative evaluation metricsIn regards to the quantitative analysis, four elements were examined. For each of these elements,

a hypothesis (in the form of null hypothesis) was made and statistically examined. These

elements, and the null hypothesis made to test them, are the following:

1. Estimation of cognitive styles by the Facebook game : The Facebook game cannot reveal players cognitive styles.

2. Prediction of user interests by the Facebook game : The Facebook game cannot predict players museum interests.

3. Success of on-site recommendations : The recommender cannot provide the best-suited exhibitions for each user

4. Prediction of visiting style by the Facebook game : Facebook game players choices of museum templates cannot reveal their preferred visiting style for

a physical museum.


13/39



4.2. Analysis of experimental results4.2.1.Facebook game results analysisGameplay choices: All participants, apart from 3 that did not have a Facebook account, had

played the MMS Facebook game. From the 9 available avatars in the game, three were neverchosen (TV persona, Diplomat, Rapper). From the remaining 6, the mad scientist was the most

popular since 7 people chose it, 5 chose the Engineer, 4 the Old-wise, 3 the Artist, 2 the Judge

and 2 the Alien. This clear preference for the mad scientist might be due to the position of the

item in the game, since it was the very first avatar in the top left corner of the table with the 9

avatars. From the available pets, the gold fish was never chosen. The dog was the most popular

with 14 people choosing it. 4 people chose the monkey, 4 the owl and 1 person chose the cat.

Again the dog was placed at the top left corner of the table with the pets. From the available

tools, the Book was also the most popular (14 choices), being placed at the top left corner of the

choice table. 4 people chose the clock, 3 the disco ball and 2 the heart. Avatars, pets and tools

are shown in Appendix B.

A clear preference for the first item in the tables with the avatars, tools and pets was observed.

Items in the top left corner of these tables were chosen significantly more than the remaining

items. In addition, 4 people also reported that they had problems selecting these items, since by

default the first item was chosen and a single click was selecting the items without an undo

option. This usability problem, together with the known preference of the western world users

for the top left corner, could possibly explain the results observed here. For this reason, any new

version of the game should handle the usability problems reported and also vary the position of

items in the top left corner of the choice tables.

Furthermore, participants were asked for the reasons they made the above choices. As it was

found during pilot testing, the pictures of avatars, tools and pets alone were not sufficient to

stress the main feature we would like people to look at. In order to increase the success rates of

the pictures chosen, a short sentence was placed under each picture to show people the main

characteristics of the item. For example, the picture of a diplomat was accompanied with the text

I like to avoid conflicts, because the concept of a diplomat might mean different things to

different people. From the participants answers it was clear that this practice was very useful and

people did read the text in order to choose avatars, tools and pets.

Hypothesizing that the different choices of avatars, tools and pets would reflect peoplespersonalities was also supported since 15/20 people explicitly stated that they chose these items

since they reflected their personalities. Some of their answers were:

Because I am strange and they reflect my personality(male, 19 years old)

Descriptions closer to my personality (female, 23 years old)

Read the description and was close to my personality(male, 20 years old)

Representative of my personality(female, 23 years old)

Reflect my wayof thinking(female, 38 years old)


14/39



Game Usability: Playing the game, most users did not face any problems. Only 4 out of 30

reported problems, mostly regarding selection issues with the avatars, navigation problems and

avatar movement problems (at times it was getting stuck). In addition, 26 out of 27 people

reported that the game was easy to play, 24 out of 27 said that they enjoyed the game and most

people managed to collect exhibits by playing, with average exhibits per player being 7. Almost

half of the users invited their Facebook friends to play the game (13/27) but only 8/27 reported

their scores on their wall on Facebook. As for ameliorations, most participants suggested that the

game graphics could be enhanced (some found them old-fashioned). Additionally, some

participants mentioned that they had navigation problems, although a map was used in the game

to show players their exact position within their museum. As for other points for improvement,

a few people said that the games were easy and they would prefer more difficult ones, while

others suggested adding a short description to each game. One participant suggested adding

tablet version of the game.

Estimation of Cognitive Style by Facebook Game:One of the main hypotheses of thepresent study was whether a social networks game can reveal peoples personality traits, such as

cognitive style. In the following table the participants actual visiting style, as it was calculated

from their interview questions is presented together with the one the game estimated from their

choices. The different letters represent the 4 dimensions of the cognitive style (the Myers-Briggs

Type Indicator), which can take 2 values each. In particular, there are 4 dimensions of two

opposite points, like Extraversion-Introversion (letters E and I in the table), SensingIntuition

(letters S and N), Thinking-Feeling (letters T and F) and JudgingPerceiving (letters J and P).

Table 3. Users' Estimated and Actual Cognitive Style

Facebook AppCognitive Style

Estimation1

Real CognitiveStyle from

interviews

NP E N T

INT E N T

NT I N F

INF I N F

NT E N T P

E T I S T J

N T I S F

1 Successful estimations are indicated in bold blue font.


15/39



ENT E N T P

EN J E S T P

T E N F J

INT I N T J

E J I S T

TJ I S T J

INT I N T P

N J I S T J

ENF E N T P

T I S T J

T J I N F P

IN T I S F P

S T E N T P

ENP E N T P

J I N F

T I S T J

ES T E N F J

I S T E N F P

S T J I N F

The missing information on the estimated cognitive style (when cognitive style could not be

entirely computed from the game, due to player actions denoted in the table above by the +

sign) was randomly filled by the application in order to assign users a cognitive profile. The

recommender was using the completed information. For the purposes of the presented analysis

the actual data, without the additions, are used to see the effectiveness of the game in cognitivestyle estimation.


16/39



The last dimension of cognitive style (Judger-Perceiver) is the only one that can change during a

persons lifetime. The other dimensions are relatively constant. This last dimension seems to be

affected by age and most people seem to start as Perceivers and end as Judgers later in life. For

this reason, together with the fact that most of our participants were young adults, some people

scored between the two categories (denoted in the table above by the * sign). In these cases, any

estimation of their cognitive style from the game is considered correct, since these participants

could handle equally well both types of information (i.e. information designed for Perceivers and

information designed for Judgers).

For the first dimension, Extraversion-Introversion, the game correctly estimated this dimension

with 69.2%. For the dimension SensingIntuition the success rate was 58.8%, for the Judger-

Perceiver it was 77.7%. The lower score was for the Thinking-Feeling dimension, which was

only 55%.

The collected data seem to partially reject null Hypothesis .

The game could in many cases correctly estimate players cognitive style, with success rates of

55% to 77.7%. The experimenters are particularly satisfied with the above result, since the game

could reach this estimation after only three choice screens. It seems that the game is certainly

moving towards the right direction and more features need to be included to make it more

accurate. From different pilot studies and preliminary results, certain correlations were found

that could be used in the next version of the game to increase its success rates. For example, a

high correlation between peoples cognitive style and music, fashion and decoration preferences

was found. In a future version, the player will be able to dress her avatar, to decorate her

museum space and choose what type of music she wants to listen while playing. All these choices

provide further information for the better calculation of cognitive style. In addition, it was also

found that different cognitive styles might have different game preferences and preferences for

museum content. In a future game version, games from different categories should be included

and players choices should be also recorded, together with their choices for different exhibits

and content.

Prediction of user interests by Facebook Game

In addition, 21/23 participants reported that their exhibits choice in the game reflected their

actual museum interests (91.3%). The recommender used information from the game and

peoples exhibits choices in order to suggest different exhibitions to different users. Peoplereported their preferences for the 8 exhibitions available at FHW, from very interesting to

medium interesting to not interesting. The number of stars provided to the users describing the

exhibitions relevance to them, was compared to the users self-reports on their interests levels

for those exhibitions. The results rejected the null hypothesis and , implying that the

developed applications adequately predicted user interests and suggested best suited exhibitions

in many cases (please, see below for more information about recommender success on-site).

From the interview answers it was clear that the game was easy to play, with straightforward

rules, engaging and most users also found it enjoyable. In particular, regarding participants

choice of exhibits, in most cases this was a very interesting process, since most reported that


17/39



these exhibits reflected their personal interests. Only one person reported that she chose items

from all the available categories and one more that did not really think about it. All other

participants had very clear arguments about their choices:

Because I love Darwin(male, 19 years old),

I like ancient exhibits(male, 20 years old),

They reflect my interests(female, 19 years old),

Because, I usually visit museums about ancient Greece(female, 23 years old), etc.

Despite the fact that for copyright issues the images were processed and were not very clear, this

did not seem to affect users choices. In fact, only 6 /30 people reported that the quality of the

images was affecting their choices. It also seems that a game that uses such features can

accurately predict peoples interests inside the museum. Although, based on the overall

observations, the null hypothesis concerning prediction of visitors interests could be rejected, acloser look revealed that this was due to certain categories. Other categories, like peoples

interest for Biology could be correctly predicted in more than 65% of the times. It seems that

possibly the quality of the images used in the game might have affected the outcome or the

methodology followed in order to collect user data, might also affect results. More specifically,

the recommender predicts interests in generic interest areas (e.g. Biology, Ancient Civilizations,

etc.) but the users rated specific exhibitions (in this case, exhibition content was made for school

children and a few users mentioned that they found them childish). In addition, the self-

reports used might have produced biased results, since users might have told the experimenter

that they liked an exhibition more than they actually did. In any case, the fact that therecommender was successful in certain cases is promising since it provides an indication that

social networks games could predict peoples museum interests. Therefore, information from

social networks could be particularly valuable to museums and curators, in order to 1) correctly

advertise different exhibitions to different users and 2) appropriately guide users inside the

museum, by increasing the quality of their experience.

Prediction of visiting style by Facebook game: From the available museum templates, most

people chose the free museum (11 people), 10 people chose the open museum and only 1 chose

the linear museum (the museum templates are available in Appendix C). In addition, most

participants reported a preference for an ant visiting style in museums, 6 preferred a butterflystyle, 4 a fish visiting style and 3 a grasshopper style. These metaphors showed the nature of the

movement. An ant visitor moves in a clear line, views almost all exhibits, spends a good amount

of time for each exhibit, pays attention to details, moves close to the exhibits and the walls,

avoids empty spaces, follows the curators suggestions and rationale. A fish visitor moves in the

centre of rooms, does not avoid empty spaces, does not pay attention to details but rather shows

interest in the larger picture, spends short time in front of the exhibits and does not stop very

often. A butterfly visitor does not follow the curators paths or a clear line in her movement,

changes the direction of the movement frequently, usually avoids empty spaces, moves close to

the exhibits, sees almost everything, looks at details, seems to be attracted by the exhibits

accessibility, is affected by other visitor traffic (environmental affordances (Gabrielli et al. 1999,Marti et al. 2001) and stops frequently. Finally a grasshopper visitor seems to have clear


18/39



preferences and views only the exhibits that interest her. Such visitors do not stop very often,

cross empty spaces and they spend a long time in front of the exhibit they choose to see. Ant

visitors need the most time to view an exhibition from all other visitors, butterfly visitor follow

in time demands, fish visitors need less time than the two above and grasshopper visitors have

the shortest visits of all (Oppermann & Specht, 2000).

The museum templates were used in the game, since it was hypothesized that they could reflect

visiting style preferences. When the museum templates choices in the game were compared to

the interview questions about visiting style preferences, it was found that visiting style was

correctly predicted from the template choices in 6 cases and wrongly predicted in 16 cases. The

results failed to reject null Hypothesis .

Regarding the choices of the museum templates in the game with the reported visiting style, it

was found that we could not accurately predict the preferred visiting style of visitors by simply

using their game template preferences. However, the nature of the images used in the game

might have affected users choices. As some participants reported during the interview, the shape

of the museum template was very important for their choices, more than the practicality of that

template. One participant mentioned,

it looked minimal(female, 24 years old),

another said, I liked the shape(male, 19 years old),

and others commented, looked more symmetrical( male, 23 years old),

looked more contemporary(male, 32 years old),

looked more spacious(female, 24 years old)

Trying to avoid the issue of image aesthetics, in a next game version, the templates will need to

be very carefully designed. In addition, it is also important to note here that this should not

count as a point of failure for the experiment, since the use of visiting styles in the on-site

experiment was decided not to be exploited in the end, due to the lack of path diversification

ability by the recommender (because of the few items to suggest aside from shows), as explained

above.

4.2.2.On-site results analysisThis section provides an overview of the analysis of the results achieved during the on-site

deployment of the BLUE experiment, and the effect that these had on the QoE of visitors. An

analysis of the QoS results experienced during the on-site experiments is also provided.

QoE: Recommender Satisfaction and Evaluation: Consequently, due to the success rates of

the recommender, 73% (11 out of 15) of participants reported that they followed the

recommendations. When asked what they would do if they visited alone, 71% (17/24)of the

participants mentioned that they would follow the recommendations, 17% (4/24) reported that

they would not follow the recommendations and only the remaining 12% (3/24) said that they

would try to combine the recommendations with their personal opinions.


19/39



Finally, people were asked to provide their overall feedback about the recommender. In general

75% (15 /20) of the participants were very positive with the use of the recommender and only a

10% (2/20) mentioned that the recommender should have more information. Here are some of

the most interesting participant comments:

It is nice to see all the available exhibitions(male, 39 years old)

If you combine it with more information it could be very useful. I like the photo feature that you can upload on

FB. Together with your comments it is a verygood advertisement(female, 38 years old)

It is quite interesting to see what was pre-recommended according to my game results(female, 25 years old)

The recommendations were mainly close to my interests. Made my visit easier and not time consuming(female,

22 years old)

It was pleasant and different from usual. Made the visit easier and more fun(male, 25 years old)

It is nice to see details of exhibitions and assist choice(male, 19 years old)

Participants were also asked if they would prefer to visit with or without the recommender. This

gives an impressive result: 82% (22/27) of the participants answered that they liked having the

tablet and the quality of their experience was enhanced. Some of the user comments were:

I prefer having the tablet to see what is available(male, 32 years old)

I enjoyed it a lot(female, 23 years old)

Both the game and the tablet made my visit more enjoyable(female, 22 years old)

I was better prepared for the visit(male, 46 years old)

Finally, three last questions attempted to discover how important different features of the

recommender were, like avoiding visitor traffic, not missing important exhibits and to have most

effective movement patterns within the museum. Participants had to rate these features from 1

(not very important) to 5 (very important). Participants provided the score 4.6 for the

importance of seeing all relevant exhibits in a museum and not miss information important to

them. Having most effective movement patterns and not wasting time by walking back and forth

was of medium importance, since the average score was 3.9. Finally, avoiding visitor traffic also

scored 3.9.

Success of on-Site Recommendations

When self-reports were compared to the recommendations, it was found that from the total of

250 recommendations, the recommender was successful in 57.9% of the times (average successrate). However, when individual categories were studied separately, the recommender was


20/39



particularly successful to suggest certain types of exhibitions like Ancient Cities (61.3%) and

Biology Darwin (65.2%). There were lower success rates for other types of exhibitions like

Biographies-Kazantzakis (42,1%) and Environmental exhibitions (57.7%). It is interesting to

note that recommender was also suggesting activities like a short break at the museum caf or

museum gift shop. These recommendations were not based on the individual profiles but only

on the exhibitions time schedules and gaps between shows. In this case, of the non-targeted

suggestions the users reported that in only 51.3% of the times they were pleased with these

recommendations. In other words, and although the sample used in these studies does not allow

a smooth statistical analysis, it seems that targeted recommendations based on individual profiles

score a lot higher than random, non-targeted ones. Certainly this is only an indication that the

present approach of creating and using visitor profiles, moves towards the right direction. With

the current results, we have a partial support from Hypothesis , since users sometimes

provided contradicting statements (i.e. the satisfaction levels reported do not always match the

stars given by the users to the different exhibitions). The reasons for these inconsistencies are

not clear, especially when the Facebook game seems to correctly reflect user interests in morethan 90% of the cases. However, the very nature of the exhibitions seems to have played a very

important role. The exhibitions were highly targeted to school students and in many cases, our

participants found the content below their expectations or childish as a few put it. In the

Discussion section below, a possible explanation is provided in greater detail. Finally, the

difference of results between interests predicted in the game and those given by the

recommender on site can be technically explained by the following: regarding interests in game,

people gave feedback on the suitability of the objects they choose to represent their actual

preferences. Contrarily, recommendations were evaluated by comparing computed ratings (on a

1-5 scale) with the users ratings, which leaves much more place for errors than with a binary

answer as was the case with the in game evaluation.

Photo Option and Online Visit Sharing: Using the tablet application, the user could take

photos from their visit and upload them to their Facebook account. Although only 39% (9/23)

of the participants answered that they would upload these pictures or that they already had, at

the end of the interview when asked about their general views of the experiment, most people

said that they really liked the photo feature and most took pictures using the tablets during their

visit.

An interesting situation was observed in regards to the photo option in the MyMuseumGuide

application. Although only 9 people explicitly mentioned that they would upload pictures of theirvisit on their Facebook account, most users used the photo option and many said that they liked

it. The finding is not contracting, since people might want to document their visit, but not

publish it on social networks, protecting their privacy. In addition, our average participant age

was 30 and older adults might have a different approach to privacy issues than younger adults

concerning information they share on social networks. Nevertheless, the photo option seems to

be useful and liked. The only addition would be to provide the option of sharing the information

on Facebook or keeping it for personal use. Facebook already provides this option and it could

be explicitly incorporated in the MyMuseumGuide application.

Tablet Usability:Almost all users stated that the tablet application was very easy to use. Only

one said that it was medium easy. All (100%) of the participants explicitly stated that the


21/39



application was enjoyable and 90% (19/21) of the users mentioned that the information was

presented in a satisfying manner. When asked if a map would be useful, 74% (17/23) of the

participants agreed. Although this was a suggestive question, it seems that a map option would

be useful. While the My Museum Guide mobile application featured a map, it was not activated

due to an unresolved problem as to how best switch floors when a user transits between floors.

To not confuse users, it was chosen not to display a map. Participants were also asked if the

recommendation notations were clear and if they would make any changes. Most participants

were satisfied with the notation system used (i.e. stars for each exhibition). One participant

mentioned that he did not realize that these stars were the recommendations and another one

said that perhaps a bar with percentage of interest would be more useful than the stars. Finally,

all participants were asked about other information they would like to have. Most frequent

requests were a map and more pictures from the exhibitions, and three participants also asked

for a feature that they could also rate the exhibitions.

Effect of Personalisation and Content Adaptation:Although all content provided to theusers was adaptive and prepared for the different cognitive styles, the users did not realize this

content adaptation. Although they were happy with exhibition descriptions, they did not realize

that their partners and friends had different descriptions and often did not open the content

pages that provided more in depth information for each exhibition. Only 6 participants said that

they checked their friends screens but did not pay any more attention. Thus, none of the users

were aware of the adaptive content in the descriptions of the exhibitions and only 6/23 checked

their friends screens and saw that there were different stars provided to different users.

Consistent with literature describing how users do not read text (Chairman & Claremon, 1977,

Falk et al., 1986), our observations confirmed that finding. It seems that adaptive content is notnecessary at least in the form of text, since users do not really read this information. However,

users did notice the pictures used for the different exhibitions and some asked to have more

pictures from each exhibition available at their tablets. This observation might imply that

adaptive content is relevant but in alternative forms to text, like audio or visual content. Further

research is required however to determine which form of adaptive content is best for different

visitors, possibly related to their individual cognitive styles.

Group Visitors Evaluations: Since most museum visitors visit in groups, it was important to

record group visitors behaviour with the tablet application. Participants were asked to describe

their behaviour within the museum and whether they did what the recommender suggested orfollowed the groups wishes. Only one participant mentioned that he would not follow the group

and continue with his personal preferences as reflected on the recommender suggestions.

Similarly, only one person said that he only followed the groups decisions. It seems that most

group visitors tried to combine different suggestions for the different group members and

proceed accordingly. However, the situation was very different for participants visiting with

children, since they all said that they only did and would do what the child wanted.

QoS evaluation:

Upon examining the list of metrics collected during the experimentation, we realised that

measures regarding QoS were not necessary for our experiments. Indeed, as we are not running a


22/39



real-time system and neither QoE measures nor the visitors experience are affected by factors

relating to QoS, measures regarding the latter could be neglected. In hindsight, evaluating

comments received by visitors, the initial assessment can be confirmed. No user expressed any

concerns about the QoS, hence, there was no need to re-evaluate the need for those measures.

4.2.3.Usefulness of cognitive style in recommendationsThe sequence of recommendations displayed on MMG was the final service proposed to guide

visitors and supported fully the user experience during a museum visit. In addition to gathering

user feedback and their satisfaction regarding the recommendations provided, we tried to

observe the influence of using cognitive style on recommendations. The Facebook game was

very important in the experiments since it provided both the cognitive styles and the exhibition-

related interests (through items chosen for the virtual museum). For users who did not play, the

personal profile was empty and the cognitive style taken randomly.

With the specific characteristics of the museum, recommendations had to follow strict rules to

ensure their usefulness. (1) The small number of exhibitions (resp. POI) implied that no filtering

was applied: all exhibitions and POIs were shown, whatever their matching score for the user. (2)

The fixed schedule of some exhibitions implied that recommendations were ordered according

to the moment they should be visited, respecting the time schedule. (3) Because some exhibitions

had very similar topics and thus had the same matching score, they were presented with a slightly

different number of stars (around the normally computed value). We postulated here that

diversity in the recommendation scores displayed to the user would help him making a choice

(which might be difficult when a set of items having all the same score are presented).The

sequence matching was computed using Cosine Similarity on the set of matching scores POI in

the sequence (details on the recommendation algorithm can be found in Naudet, 2013). We haveanalysed user satisfaction regarding the recommendations to assess the accuracy of computed

sequence matching scores.

A first statement was that user profiles with only a cognitive profile (CP) (considering only the

cognitive style estimation derived from the game without other factors like choices of exhibits,

etc.) did not contain enough information to provide relevant recommendations (a complete

profile includes information about cognitive style together with players exhibit choices in the

game, which provided a more complete picture on possible museum preferences). Indeed the

analysis conducted before the trials to map cognitive styles to museum exhibitions topics

revealed only two rules that have been exploited: (1) people with cognitive styles containing thefeeling dimension (F) usually like literature and biographies; and (2) people with cognitive styles

that contain the thinking dimension (T) usually like natural sciences. According to the topics

associated to the museum exhibitions, 4 at most could match the (T style, with only one

matching a CP containing both the (F) and the (T styles, as shown in the table below:


23/39



Table 4: opics associated to exhibitions, corresponding to cognitive style dimensions.

Exhibition Literature and bibliographies Natural science

Cognitive Style dimension Feeling (F) Thinking (T)

Darwin

Earth is our home -

Ice Worlds -

Your planet needs you! -

Other exhibitions - -

On the whole set of experiences, the two visitors for which we had only the CP rated Darwin

high, which corresponds to their supposed behaviour according to their cognitive profile that

contains feeling (F). However they also liked other things, thus nothing could be concluded.Knowing this, having in the user profile also personal interests complementing CP-based

interests was mandatory for the conducted experiments, in order to get recommendations that

are sufficiently diverse.

The synthesis showed that almost all visitors were satisfied by the recommendations and have

followed them, whatever the matching score of the sequence they were proposed. The visitors

that did not follow the recommendations did it for reasons independent from the

recommendation quality (e.g. following a childs decisions). This tends to show that the

computed scores were relevant.

Finally, it was interesting to observe that while sequences matching scores are low when only the

cognitive profile prevailed (i.e. no or just a few personal interests are known), they rise to

medium or high as soon as the number of times personal interests prevailed in the computation

of exhibition matching scores is higher than for the CP. Since we have shown that only few

exhibitions match with cognitive styles mapping rules, we have the obvious result that the richer

(or the more diverse) the profile, the better the chances to make a relevant recommendation. In

conclusion, the experiments did not allow showing the usefulness of using visitors cognitive

style in a recommendation process.


24/39



5.EXPERIMEDIA component assessment5.1. EXPERIMEDIA Experiment Content Component (ECC)

The ECC was used to automate data collection in a secure and centralized location in Greece.

The component was integrated with our systems as described in Sections 2.3 and 2.4 ofdeliverable D4.8.2. The deployment of the RabbitMQ messaging bus proved a bit tricky but

doable with a hand from IT Innovation staff. The same held true for switching the system to

operate exclusively using SSL. After the installation of the server-side ECC, the infrastructure

was briefly tested with the provided sample implementations.

Due to the client interacting with the server-side ECC being based on Android, the first version

of the libraries had some compatibility issues. However, IT Innovation was able to provide an

Android implementation of their client libraries in a short time period and we could proceed

with the implementation.

The implementation of the ECC running for our experiments was pulling data every four to five

seconds from all connected devices. Due to improvements in the ECC design and experiment

state handling, we were able to allow for dynamic client connections, drops, and reconnections;

all transparent to the user. We were happy with the performance and like to commend the quick

response time and uptake on proposals for improvements by the team at IT Innovation.

As of now, the only point where we were not completely satisfied is the support for data analysis.

As the ECC compiles a humongous amount of data, it is rather cumbersome to prepare the data

for subsequent analysis. Also, due to the complexity of the data model, extracting the data from

the database in a usable format also proves rather cumbersome. However, these points have

been noted by IT Innovation and will be subject to further analysis and improvements.

5.2. EXPERIMEDIA Social Content Component (SCC)The SCC sub-component used in the My Museum Guide Android application is the Social

Integrator (SI). It allows us to propose users to log into their social medium of choice and,

thereby, connect him through our application to his social environment. The SI was thrown into

disarray by an adjustment of the Facebook API. However, the NTUA team was able to quickly

address the issue. As of now, the SI performs admirably and, without further change to the

Facebook API, should continue to do so.

A proposal for the development of a client-side monitor of social interaction was taken into

account by the development team and proposed as a web-service this summer. We were able to

deploy the server-side of the social web-service but, unfortunately, development of our own

application had proceeded to a point where it would have been a non-negligible risk to attempt

integration. However, we are confident that future EXPERIMEDIA partners will find the

development useful as it complements the ECCs EM nicely when it comes to collecting social

data.


25/39



6.DiscussionThe two applications developed within the framework of EXPERIMEDIABLUE were tested

with real users at the FHW in four separate sessions. The sampling processes replicated the

processes the museum will apply in order to attract new visitors, advertise exhibitions and alsocollect relevant information for user profiling, since the museum will also its Facebook page to

host the game and invite new players/potential visitors. Having used Facebook for sampling

purposes, the networks efficiency was also observed since people responded.

Users found both applications to be highly usable and enjoyable. As users also mentioned, it was

very interesting to link peoples personalities and actions prior to the visit with the physical

museum visit. The novel approach of using a social networks game for museum visitor profiling

was successful in numerous ways but most importantly because it opens a road towards the

exploitation of the vast quantities of information available in social networks and its use in

adaptive technologies. Overall, visitors QoE has been enhanced and their final impression withinthe museum was very good. Roughly (see previous section for exact numbers), 82% liked using

the tablet with MMG, 75% had a positive impression using the recommender and 73% actually

followed the recommendations. Those results, despite an average precision in recommendations

of 58% (on exhibition ratings, on a 1-5 scale) are very encouraging knowing the constraints

induced by the museum venue and the little amount of data exploited for profiling users. With

the MMS, we were able to show with a high success rate (resp. 91.3% and 67.17%) that user

interests and cognitive style could be predicted from a simple game targeting museum topics.

In the following sections we discuss the results of data analysis regarding our four hypotheses.

H1: The Facebook game can reveal players cognitive styles. The data collected in the present study

provided partial support for the above hypothesis. As discussed above, this first version of the

game only explored the possibilities of a game that could reveal cognitive styles. There were clear

tendencies towards this direction that definitely require further development. The game only

used three screens to calculate players cognitive styles. From the available results from our

previous studies (preliminary results from pilot studies), we know that different features that can

be added to the game can provide further information and possibly more accurate results. The

main challenge for designing the game was to identify the key pictures to best reflect the

different dimensions of the cognitive style. Keeping in mind that for the proper estimation of

cognitive style, a highly trained psychologist should interview a person over a significant amountof time, the game had to significantly accelerate this process by reducing it to item selection in 3

screens. Considering the above, the results were more than satisfying and promising for further

development. In addition, this is a highly novel approach, not simply because it attempts to

estimate users cognitive styles from a simple game, but also because information from social

networks can be used in a user profiling process in adaptive systems, whether these will be used

in cultural heritage, education, etc.

H2: The Facebook game can predict players museum interests.The Facebook game was able to correctly

predict players interests and support the 2nd Hypothesis. Most users reported that they chose

items that reflect their personal interests and not simply because of item aesthetics or imagequality. Information deriving from social networks can be particularly valuable to different


26/39



professionals like teachers and curators. Knowing users interests information can be tailored and

directed accordingly. In particular, museums can use this information to target specific groups in

order to advertise different exhibitions but also in order to support the physical museum visit,

significantly enhancing the quality of visitors experience.

H3: The recommender can provide the best-suited exhibitions for each user. Based on the game data, therecommender was indeed successful in providing the most appropriate exhibitions to the

different users but for some and not all exhibitions. So far, my museum guide has been designed

for single visitors, focusing on individuals and single cognitive styles. However, as known from

the literature but also observed during our experiments, visitors usually visit in groups. The

recommender could significantly increase its success rates when it will combine the information

of the individual cognitive styles and the information about the group member in a single visit.

For example, exhibition 1,2 and 4 will be the best for user X. Knowing that X is visiting together

with Y and her best options would be exhibitions 1,2 and 5, the recommender can suggest

exhibitions 1 and 2, informing the users that these would be the best choices for their group.Especially, for families visiting, the recommender could calculate the cognitive styles of the

adults and the age and gender of the accompanying children, in order to suggest best options.

These future additions of the recommender would hopefully make it suitable for group visits,

targeting a known museum technology problem that of group visits and content provided.

H4: Facebook game players choices of museum templates can reveal their preferred visiting style for a physical

museum.The above hypothesis was not supported with the available data. The main reason seems

to be the aesthetics of the images used, since certain images seemed to attract the majority of

preferences. Attempting to capture the preferred visiting style before ones visit seems to be a

very demanding task and alternative ways should be further explored. One possibility would beto use the known link between cognitive and visiting style (Antoniou, 2010). Once the cognitive

style is known from the Facebook application, an estimation of possible visiting style preferences

can be derived and later used at the museum. This is a field for future exploration that remains to

be studied.

The EXPERIMEDIABLUE team faced numerous challenges during the design, implementation

and testing of the applications, mainly due to the nature of the venue. FHW was not a traditional

museum hosting object-based exhibitions. In addition, the content of the exhibitions was highly

targeted to school students. The information was simplified and generic, mainly in the form of

3D films. Only one exhibition was using interactive technology to present environmental issues.The layout of the museum and the nature of the exhibitions (i.e. highly targeted, film form,

specific time schedule) have affected the outcome of the experiments, since none of the users

was of school age. Most users were adults invited through Facebook and they found the

exhibition content of a school level (some called it childish). This had enormous implications in

the present work, since although someone might like Biology for example, they could give the

Darwin exhibition a low mark because it was not of the desired depth. Combined with the fact

that most users did not access, read or even notice the adaptive content, possibly adaptivity at

this level might not be meaningful. Post-adapting a strongly focused exhibition (as these

"exhibitions" were focused on children) does not have an overall effect (positive or negative) on

the visitors' experience. Adaptation is something to be considered beforehand, to cater fordifferent target groups and different views/interpretations of the exhibition's message. Altering


27/39



text messages does not have a strong effect on the experience. However, recommendations are

certainly useful. Given that the museum is a small one with a lot of limitations on the visit

(hours, presentations, ticketing policy) the overall visitors' feedback is positive. For a large

museum, with lots of exhibits (more than what an average visitor would be able to view in a day's

visit) and more freedom of movement the impact would have been even greater. In larger

museums, such a system would be particularly useful, since it would allow curators to know

visitor interests even before their visit, could suggest routes inside the museum to avoid visitor

traffic and provide opportunities for both visitors and museums to connect the visit with social

networks.

In the present work, efficient recommendation depended on two main things: the quantity of

information contained in both the user profiles and the items to be recommended; and the

matchmaking algorithm that actually computes the items to recommend. In the case of our

experimental context, the following issues emerged:

Insufficient number of topics attached to each exhibition; Insufficient user profiles: items in the FB game mapped to one topic only and the

number of different items was not enough;

Number of topics used to characterize both POI (in particular exhibitions) and userinterests was clearly not rich enough: a multi-topic classification of exhibition should

have been used;

Only two rules mapping cognitive styles to exhibitions is not enough, making therecommendations based on cognitive styles not necessarily relevant;

There were not enough items to recommend; There were only few participants, but this is not a factor impacting the precision of

recommendations.

For future research, the following important points will have to be taken into account. To see

the effect of knowing the cognitive styles on recommendations, we have to (1) know the users

cognitive profile and (2) have a significant set of rules mapping each cognitive style to interests

related to exhibitions topics. Then, to be able to increase recommendation accuracy, we need to

have: (1) more interests deduced from MMS; (2) exhibitions (resp. POI) described by a set of

topics instead of only one. In summary, the profiling process through the MMS game only was

too weak to get accurate results. The average results that we have tend to show nevertheless that

we should carry on investigating our approach because in the end people were satisfied by the

recommendations.

In general, EXPERIMEDIABLUE attempted to combine cultural heritage, social networks and

peoples personalities in a unique way. The academic community recognized the novelty and the

significance of the approach, since parts of the present work have already been presented in

international conferences and published in conference proceedings. A new door opens in the

exploitation of the available information in social networks for adaptivity purposes. The uses of

such practices can be numerous and remain to be studied. Moreover, the users found the

approach engaging, entertaining and promising. The popularity of social networks and social

network games make it an excellent field for use in cultural heritage. Finally, the present study


28/39



showed that certain improvements of the two applications (My Museum Story, My Museum

Guide) could significantly improve the quality of the visitors experiences by:

1) predicting cognitive styles more accurately2) providing even better recommendations3) combining existing information and individual profiles with group visits and family visits


29/39



7.ConclusionEXPERIMEDIABLUE was an attempt to connect physical space attributes, human

personalities, and social networks. The experiment investigated novel ways to extract museum

visitors profiles and use them in order to provide personalized information. Viewing themuseum visit as a process that starts before the actual visit and finishes long after that, social

media were employed to provide a continuation of the visit, in digital space. The visitor could

use material from their visit to create personal diaries of the visit, publish them on social media

and share the experience with friends.

During the trials at the Foundation of Hellenic World in Athens, Greece, we observed the

interactions of visitors with the MyMuseumStory (MMS) game and with the MyMuseumGuide

(MMG) application. Furthermore, visitors were asked to fill in a questionnaire in order to gather

feedback on their Quality of Experience and the usefulness of the different personalization and

recommendations they were provided. Experiments revealed that a good proportion of visitors(over 70%) liked and followed recommendations, had a positive impression of using the

recommender and reported the mobile guide enhanced their quality of experience. While we

were able to show that the Facebook game could predict user interests and cognitive profile for a

small sample of 30 persons, the overall precision of resulting recommendations was around 60%

and although they have been happy with provided Point of Interest descriptions, we do not

know if the cognitive profile-specific adaptation is responsible for this. However, these results

are encouraging regarding the small amount of data available to make the recommendations (e.g.,

we could only use a few rules to map cognitive styles to interests).

Overall, visitors feedback shows that BLUE tools improve the quality of visitors museum

experience, and the interest in individualization of visits and personal technology-enhanced

guiding. Moreover, the introduction of social gaming as an extension of visits trough social

network has received positive feedback. From a scientific perspective, the usefulness of gaming

to derive visitors interests has been proven, while promising results have been obtained

regarding cognitive profile detection through gaming and its use for visit personalization.

Quoting an experiment participant:

I really liked the experiment. It gave me the opportunity to see various

exhibits that piqued my interest. I believe that they were based on my

character. They suited me a lot. It was a nice experience. There was also the

application, which was very important, as I could go through everything and

not miss anything. And it was very interesting. [sic]

BLUE opens a very promising road. The vast numbers of social network users implies that there

might be immense data available for exploitation; data that could be directly used for the creation

of personalized applications for spaces of different characteristics, like museums. To this end,

BLUE can help in the introduction of a whole new set of social and networked media

experiences.


30/39



8.ReferencesAntoniou, A. (2009) A Methodology for the Development of Museum Educational Applications:

visitor inspired museum adaptive learning technologies. Doctoral Thesis, University of

Peloponnese.

Antoniou, A., &Lepouras, G. (2010) Modellingvisitors profiles: adaptation for museum learning

technologies.ACM Journal on Computing and Cultural Heritage, 3(2), 1-19.

Gabrielli, F., Marti, P., Petroni, L., 1999. The environment as interface, i3 Annual Conference:

Community of the Future. Siena, Italy, p. 44-47.

Marti, P., Gabrielli, F., Pucci, F. Situated Interaction in Art. Personal and Ubiquitous Computing

2001; 5: 71-74.

Naudet, Y., Lykourentzou, I.,Tobias, E., Antoniou, A., Rompa, J., Lepouras, G., (2013), Gamingand Cognitive Profiles for Recommendations in Museums, Semantic and Social Media

Adaptation and Personalization (SMAP) workshop 2013, Bayonne, France, Dec. 2013.

Oppermann, R., Specht, M., 2000. A Context- sensitive Nomadic Information System as an

Exhibition Guide, Ubicomp00. Bristol, UK, p. 127-142.


31/39



Appendix A. Semi-structured interviewDate:

Participant #:

Name:

Age:

FB user name:

Occupation/ educational background:

If in a group, who had the tablet (additional information about the number of group members

and other details):

The Facebook game

- Did you play the FB game? If not, who played it? Is this person visiting with you?

- Which avatars, pets and tools did you choose?

-Did you have any problems choosing avatars, pets, tools?

-Why did you choose these particular avatar, pet and tool?

-Which museum layout did you choose?

-Why did you choose this particular template?

- During a typical museum visit do you:

a) I usually see most exhibits, most of the times in a serial order (in a line). I also move close to the

exhibits, trying to see details. ____

b) I usually see most exhibits, most of the times in a non-linear fashion (not following a clear line). I also

move close to the exhibits, trying to see details.____

c) I usually see most exhibits, since I mainly moved in the center of rooms, trying to get the general

picture. ____d) I usually see only the exhibits which are particularly interesting to me.____


32/39



- How many exhibits did you collect?

-Did you have problems choosing exhibits because of the quality of the images?

-Why did you choose these particular exhibits?

- Did you invite any of your friends to play the game?

- Did you publish you score on FB?

- Was the game easy to play?

- Was the game enjoyable?

- What features of the game did you like and dislike?

- Can you suggest any improvements?

- What is your opinion about cultural institutions using games to engage future visitors?

- Are there any circumstances that the above practice would not be desired/ acceptable?


33/39



Inside the Museum: The mobile app recommender

- How interesting did you find the recommendations? Were they recommended to you?

Not at all

interesting

Totally

Interesting

Was it recommended

to you?

1 2 3 4 5 YES NO

Darwin

Ancient Miletus

Ancient Agora

Ancient Priene

Antartica

Kazantzakis

Your Planet

Needs you!

Ice Worlds

Cafeteria

Gift shop

Take a photo

- Did you mainly follow the recommender suggestions? What were the reasons?

- Would you upload photos and a diary of your visit to FB?

Tablet Usability

- How easy was it to use?

- Was it enjoyable to use?

- Was the information presented in a satisfying manner?

- What other information would you like to have?


34/39



- Would you have made any changes in the recommendations notations (the number of

stars) that were provided? Why?

- Would a map have been useful to help you find exhibitions?

General questions

- Who did you visit with?

- If you visited in a group, what did you do, what the group wanted or what the recommendersuggested?

-If in a group, did you check other members screen? What did you notice?

- If you were alone, how likely is it that you followed the recommender suggestions?

- Overall, what were the main advantages and disadvantages of the recommender?

- In visiting a museum, which of the following are important to you (rate from 1-5)?

Totally

Disagree

Totally

Agree

1 2 3 4 5To see as many exhibits that I might find

interesting as possible

Not to waste time by walking back and

forth

To avoid visitor traffic

- What is the minimum and maximum time that you could have spent today in the

museum?

Minimum time (minutes):

Maximum time (minutes):


35/39



- In the end, do you think you would have enjoyed the museum visit the same way,

better or worse without the tablet and/or without having played to the MyMuseumStory

Facebook game?

Please, answer the following questions. BOTH lists are equally valuable. Try to answer as your

really are, not how you may wish you were, or have to be at work. Tick the sentence that best

describes you most of the times in your everyday life. Either tick the sentence on the right or on

the left.

I have high energy I have quiet energy

I talk more than listen I listen more than talk

I think out loud I think quietly inside my head

I first act, then think I think, then act

I like to be around people a lot I feel comfortable being alone

I prefer a public role I prefer to work behind the scenes

I can be easily distracted I have good powers of concentration

I prefer to do lots of things at once I prefer to do one thing at a timeI am outgoing and enthusiastic I am self-contained and reserved

S N

I focus on details I focus on the big picture

I admire practical solutions I admire creative ideas

I remember facts I only notice new things

I see how things are I see how things could be

I live in the here and now I plan the future

I trust actual experience I trust my gut instinct

I work at a steady pace I work in bursts of energy

Learning new skills is tiring I enjoy learning new skills

I want clear instructions I prefer to figure things out

T F

I make decisions based on facts I decide based on my values and feelings

I appear cool and reserved I appear warm and friendly

I am direct I am diplomatic and tactful

I prefer honesty and fairness I prefer harmony and compassion


36/39



I take few things personally I take many things personally

I am motivated by achievement I am motivated by appreciation

I enjoy debates I avoid conflicts

J P

I make decisions pretty easily I have difficulties making decisions

I am serious and conventional I am playful and unconventional

I am seldom late I am usually late

I work first and play later I play first and work later

I see the need for most rules I question the need for many rules

I like to make and stick with plans I like to keep plans flexible

Thank you for your time!


37/39



Appendix B. Avatars, pets and tools IconsB.1. Avatar icons

The different avatars represent different

ends of the cognitive style dimensions.

The mad scientist uses the stereotypical

view of an introvert individual, lost in

his own studies. Th

D4.8.3 BLUE Experiment Results and Evaluation v1.0

Documents

Transcript of D4.8.3 BLUE Experiment Results and Evaluation v1.0