D4.10.3 SmartSkiGoggles Experiment Results and Evaluation v1.0

7/27/2019 D4.10.3 SmartSkiGoggles Experiment Results and Evaluation v1.0

1/42

This document describes the final evaluations of the experiment "Smart Ski Goggles". Itfurthermore contains detailed information about the methodology and the resulting insights.

Additionally, the technological implementation and the generation and discussion of businessmodel scenarios is described.

D4.10.3

Smart Ski Goggles Experiment Results and

Evaluation

2014-05-31

Gerald Binder (evolaris next level)

www.experimedia.eu


2/42

EXPERIMEDIA Dissemination level: PU

Copyright evolaris next level GmbH and other members of the EXPERIMEDIA consortium 2014 1

Project acronym EXPERIMEDIA

Full title Experiments in live social and networked media experiences

Grant agreement number 287966

Funding scheme Large-scale Integrating Project (IP)Work programme topic Objective ICT-2011.1.6 Future Internet Research and Experimentation

(FIRE)

Project start date 2011-10-01

Project duration 36 months

Activity 4 Experimentation

Workpackage 4.10 EX10: Smart Ski Goggles

Deliverable lead organisation evolaris next level GmbH

Authors Gerald Binder (evolaris next level GmbH)

Reviewers Stefan Prettenhofer (Infonova)

Version 1.0

Status Final

Dissemination level PU: Confidential

Due date PM32 (2014-05-31)

Delivery date 2014-05-31


3/42



Table of Contents

1. Executive Summary ............................................................................................................................ 4

2. Introduction and Experiment Objectives ....................................................................................... 5

3. Methodology: Co-Creation Process ................................................................................................. 7

3.1. Focus Groups ............................................................................................................................ 7

3.1.1. Setup ....................................................................................................................................... 7

3.1.2. Results ..................................................................................................................................... 8

3.2. Online Survey .......................................................................................................................... 10

3.2.1. Setup ..................................................................................................................................... 10

3.2.2. Results ................................................................................................................................... 10

3.3. Pilot 1 ........................................................................................................................................ 13

3.3.1. Setup ..................................................................................................................................... 13

3.3.2. Results ................................................................................................................................... 14

3.4. Pilot 2 ........................................................................................................................................ 16

3.4.1. Setup ..................................................................................................................................... 16

3.4.2. Results ................................................................................................................................... 16

3.5. Discussion ................................................................................................................................ 19

4. Software Development .................................................................................................................... 21

4.1. System Architecture ................................................................................................................ 21

4.2. Client App ................................................................................................................................ 22

4.2.1. Features ................................................................................................................................ 23

4.2.2. Logging ................................................................................................................................. 26

4.3. Gateway App ........................................................................................................................... 27

4.4. Backend Server and CMS ...................................................................................................... 27

4.5. Integration of Experimedia Components ........................................................................... 284.5.1. Integration of ECC ............................................................................................................. 28

4.5.2. Integration of SCC .............................................................................................................. 30

4.5.3. Integration of AVCC .......................................................................................................... 32

5. Business Model Generation and Exploitation ............................................................................. 33

5.1. Business model generation .................................................................................................... 33

5.1.1. Methods ................................................................................................................................ 33

5.1.2. Results ................................................................................................................................... 34

5.2. Discussion ................................................................................................................................ 37


4/42



6. Dissemination ................................................................................................................................... 39

7. Conclusion ......................................................................................................................................... 41


5/42



1. Executive Summary

This deliverable presents the finals results of the Smart Ski Goggles experiment at Schladming. Itprovides information focused on the three major parts of the project. First, we describe our

methodological approach based on co-creation to develop a system displaying real-timeinformation in ski goggles equipped with a micro-display. We will discuss the results and explain

why we think this approach was suitable for such kind of service. Second, we explain thetechnological implementation of Smart Ski Goggles and the integration of external services as

well as the integration of the EXPERIMEDIA baseline components. We discuss thetechnological hurdles and will summarize that from a pure technological perspective the service

could be implemented in real-life. Third, we provide information about possible business modelscenarios for Smart Ski Goggles. Here, the summary is not definite. On the short-term and ifsuch a service is solely focussing on data ski goggles it won't finance itself as the market potentialis too small at the moment. But on the other hand a ski resort could use this service for

marketing reasons. Furthermore, parts of the services features could be used within smartphoneapps or info terminals. The experiment fulfilled the venue-oriented goals of the experiment togive the venue a good indication on whether real-time information on data ski goggles would bea service their guests like and to provide the venue with business model scenarios to decide how

such a service could be operated. Apart from that the other major experiment goal to researchhow a real-time information system could be implemented into a wearable data ski goggle from atechnical and feature-oriented perspective was reached and the results are discussed in thisreport.


6/42



2. Introduction and Experiment Objectives

This document describes the results of the experiment "Smart Ski Goggles". It describes notonly the integration of EXPERIMEDIA software components, but also the general

implementation of the software as well as the co-creation process within the experiment and theway how business scenarios and exploitation ideas were generated.

The experiment Smart Ski Goggles aimed at the experimentation of a real-time information

system implemented into a wearable data goggle (Oakley Airwave). The displayed informationabout lifts, slopes, weather, hospitality, community activities and the resort in general supportusers with congestion monitoring and basic navigational hints. Smart Ski Goggles is aiming to

enhance the visitor experience while skiing on the mountain.

The experiment builds upon the outcomes of a field study conducted during the FIS Alpine

World Ski Championships 2013 by the competence centre for mobile communication andinnovation evolaris. The experimentation will implement an application for the Oakley Airwavedigital data goggle displaying and processing information, which was found most useful by skiersin a real-life test. Thus the Smart Ski Goggles app will integrate real-time information aboutcurrent load and basic navigation for lifts, slopes and hospitality points of interest in the ski area,as well as social media features. Current temperature, actual weather forecasts and avalanche

warnings will be implemented in the app to keep the users well informed about the currentconditions on the slope.

The experiment integrates the following EXPERIMEDIA software components:

ECC for experiment control & monitoring

SCC for acquiring a filtered Twitter stream

AVCC for image analysis of visual data of cameras

Evolaris as active member of the European Network of Living Labs (ENoLL) followed a co-creation approach in the app development and applied a design science methodology to derive atailor-made solution right from the end users' needs and requirements as starting point. A

targeted dissemination and exploitation strategy (including business model scenarios) understrong involvement of the local stakeholders and business partners in the Schladming resort at all

stages of the experimentation will guarantee to meet the expectations and assure a maximum ofsustainability of the proposed solutions, also beyond the projects lifetime. This is underlined bythe composition of LoS partners (resort operator, local retail expert, tourism organization andspecialized technology provider).

From a pure technical perspective the experiments goals are:

to experiment with a real-time information system implemented into a wearable data skigoggle;

to understand the practical usage potential of real-time information displayed in smart

glasses;


7/42



to investigate an emerging technological solution, bring the user in at an experimentalstage of an upcoming commercial solution.

The general objectives of this experiment are to:

analyse and define the stakeholder needs in the Schladming venue & the user needs/requirement profile for a Smart Ski Goggles app;

implement a real-time information system for a wearable ski data goggle (OakleyAirwave) in the Schladming international ski resort;

pilot the developed app in real-life settings, disseminate the results and define suitablebusiness models to assure long-term sustainability of project results.


8/42



3. Methodology: Co-Creation Process

A co-creation approach is being applied to integrate all the relevant actors and stakeholders(users, venue operator, technology provider, etc) into the conception, implementation and

evaluation of this experiment. To maximize the impact of the proposed technical solutions end-users and stakeholders are continuously involved in the development process.

The recruiting of the participants was done via our Living Lab database, where hundreds of

potential participants for studies are stored and our social media as well as other communicationchannels (e.g. poster, flyer). Additionally, our local partners at Schladming helped acquiring theparticipants. For the online survey (step 2) we used a commercial service providing a

representative sample of the Austrian population.

All four steps of the co-creation process are tightly linked together to gain a maximum of valid

insight.Figure 1 shows an overview of the different steps and their timing.

Figure 1: Co-Creation timeplan

3.1.

Focus Groups

3.1.1. Setup

The focus groups should basically answer the question: What features should beimplemented and how?

Thus, two focus groups were conducted to discuss user requirements, screen designs and

interaction concepts. This was the basis for the conception of the Smart Ski Goggles softwareand a representative online survey (the second step in the Co-Creation approach). Each of thefocus groups consisted of seven participants and lasted for around 120 minutes.


9/42



3.1.2. Results

The most wanted features were lift waiting time, navigation, notifications (warnings), generalresort information, as well as weather information and forecast.Figure 2 shows the outcome of afeature idea discussion in focus group 2.

Figure 2: Mindmap of feature ideas

Before we showed the used ski goggle "Oakley Airwave" to the participants, we asked them todraw how they would like to have information displayed in a ski goggle (Figure 3). There werealso ideas about Augmented Reality information directly projected into the goggles "glass". Ofcourse, we had to tell the participants afterwards, that the used ski goggle has got a built-in

micro-display. So, we asked them to think, how the GUI elements should be arranged in thatsmall display.Figure 4 shows some outcomes.


10/42



Figure 3: Sketch of information position in goggle

Figure 4: Ideas for positioning of GUI elements

In general it can be concluded that context is very important factor for software running in smartglasses. For example, a participant brought up the idea that the screen is changing to a screen

with reduced content as soon as the user is moving on the slopes. This was discussed in detailand finally also implemented. Important alerts should be visually highlighted in the GUI.

Furthermore, personalization and the possibility to configure the software to personal needswere requested features. Finally, we decided that such features could be part of a future software

version but are not essential for providing real-time information in an experimental setup.


11/42



We also asked the participants to evaluate whether and how the menu and status elementsshould be arranged. It finally turned out that a combined menu and status bar is ok.

3.2. Online Survey

3.2.1.

Setup

The online survey should basically answer the questions:Which features would you use andwhen?and How much are you willing to pay to rent or buy a smart ski goggle?

This second step examined the user requirements on a representative level and served as a basisfor a detailed target group specification for the pilot runs. We used computer assisted webinterviews with a for Austria representative sample of 1005 participants. The target sample

(people who use smartphones, have downloaded apps and were skiing at least once in the last 2years) was 382 people. The analysis was segmented by target groups "Sport", "Spass" ("Fun")

and "Genuss" ("pleasure") as well as by age and gender.

3.2.2. Results

One result of major importance is the data shown in Figure 5. We asked which feature is

interesting during skiing or in a break, as well as, whether a feature is not interesting at all. A veryinteresting detail is that the speed was almost evenly evaluated as interesting and not interesting.

Apart from that the survey confirmed our expectation that navigation, lift waiting, weatherinformation and warnings are top requested features. Additionally, it clearly shows that showingthe pulse in the screen (an idea of the early concept phase) is of comparably low interest.Showing text messages, e-mail and other social media contents was also rated as not interesting.

Figure 5: How interesting is a certain feature during skiing or in a break?

Asked which contents notifications pushed from the operation into the system should have, theparticipants who found notifications interesting in general wished most warnings andinformation about weather changes, seeFigure 6.Again, it showed that information from other

common communication channels (telephone calls, e-mails, text messages, Facebook postings)are of much lower interest.


12/42



Figure 6: Which contents should notifications have

The analysis of the target groups interest in using features like warnings, navigation and liftwaiting time during skiing, during a break or not at all showed, that the groups "Spass" ("Fun")and "Sport" are most interested in using these features, see Figure 7. The group "Genuss"("Pleasure") was ok with warnings, but everything else was clearly less interesting for it. Thisdifference is also reflected in the basic interest in buying a smart ski goggle, see Section 5 andFigure 32.

Figure 7: Interest in features segmented by target groups

In general, 63,4 % of the surveys target sample are very interested or interested in using a smartski goggle, seeFigure 8.There is a clear majority gender-wise. Men are more likely to use it (68,6%) than women (57,4 %). Almost all age groups are evenly interested, a bit surprising was the

value of 66,1 % in the age group of 50-65 years.


13/42



Figure 8: How many people are interested in using a smart ski goggle

Asked about the price the participants would be willing to pay for a smart ski goggle with thesefeatures the average was 160 Euro. On the one hand this is much higher than the price foraverage ski goggles (below 100 Euro), but on the other hand it is still very much less than the

current price of the Oakley Airwave (650 Euro).The average price the participants would be willing to pay to rent a smart ski goggle for one daywas 11,30 Euro, seeFigure 9.Very interesting details are that women and people between 15 and29 years old are much more likely to pay more than 15 Euro a day.

Figure 9: Interest in rental of a smart ski goggle and price per pay


14/42



The results of this survey were used to further improve the used software for Pilot 1 and Pilot 2

and of course for the generation of business model scenarios.

3.3.

Pilot 1

3.3.1. Setup

The focus for Pilot 1 was on usability aspects. At the same time the test runs during Pilot 1 werealso used to evaluate whether the implemented features were considered to be as useful as theresults of the focus groups and the online survey suggested.

Pilot 1 was conducted in Schladming on two days with in total 15 participants. We used theThinking-Aloud method combined with observation and interviews. The total duration of thetest for one participant was around three hours. At the beginning of the test the participants

were briefed about the system setup und provided features, seeFigure 10.

Figure 10: Briefing of participants (Pilot 1)

It was not necessary to install the Gateway App on the participants' smartphones as we could useour own company-owned smartphones. After the briefing the participants were equipped withthe ski goggles and could use it freely for the whole test run, see Figure 11.At the end of thepilot run the users were interviewed to gain qualitative feedback.


15/42



Figure 11: Testrun on the slopes (Pilot 1)

3.3.2. Results

The results of the focus groups and the online survey regarding the feature set were basically

confirmed. As expected the features lift waiting time, weather (current and forecast), informationabout huts and navigation were rated as most useful. The notification feature could not be testedextensively due to technical reasons. So, it could only one message be sent to the participants(drink voucher), but this was rated as a positive feature.

The feature lift waiting time could only be tested usability-wise, as the installation and integrationof the video cameras and the access to turnstile usage date could not be finished in time due to

organizational and resource reasons. But still, the feature was rated as very useful.

The weather feature was rated as making the impression of a completely finished feature and

should be kept like it was. The Twitter stream feature was not rated that good, as it was not clearfor many participants at all, that this would be a filtered Twitter stream and what the differenceto the notification feature was. It was suggested to use the Twitter logo at least.

Most interesting was the feature navigation, because on the one hand the participants found itvery useful and liked the feature very much. On the other hand there were still many routingerrors in the navigation (which made many navigation results useless) and also the visual

representation of the routing information was not good enough to clearly give helpful navigationhints. The former point is out of control of the software Smart Ski Goggles, but we informed theowner of the navigation algorithm and data about errors we found and some of them were fixed.

The latter point was tackled by us for Pilot 2 with many improvements in the visual presentationof routing information, e.g. with using static arrows and automatic photographic hints.


16/42



From a usability perspective we got a lot of useful feedbacks from the participants. Most of thewere related to contrast (seeFigure 12), used colours, font sizes, where GUI elements are placed

and how the interaction in lists should work.

Figure 12: Low contrast for selected list element

The Pilot test showed that usability aspects even more crucial in smart glasses software than inconventional desktop software. The reasons are manifold. First, the display size and resolution is

limited. This leads to necessary reductions of labels, which in consequence makes it harder forthe user to understand the displayed data. Second, the interaction possibilities are limited. On theremote there are only six buttons: left, right, up, down, enter, back. Actually, this makes the workon an interaction concept harder and some useful interaction principles could not beimplemented (e.g. free selection of menu item). But, on the other hand due to these limitationsthe final interaction concept was easy and clear to the participants. Third, the frame conditions

when using the smart ski goggle have to be considered. The display has to be readable also withbright sunlight and even during moving on the slope. Especially the second case is important.

The user must not be forced to focus his eye longer than a fraction of a second onto the display

and away from the slope. So the displayed information must be very easy recognizable. Another

idea would be to fully switch off the display as soon as the user is moving faster than a definedspeed.

Figure 13 shows an overview of recommendations to improve the usability of Smart Ski Gogglesresulting from the insights we could gain during Pilot 1. Many of them were implemented intothe software for Pilot 2.

Figure 13: Recommendations to increase usability

Priority Effort Recommendations

Mittel Niedrig nderung Kontrast ListenelementMittel Niedrig nderung Kontrast Farbehinterlegung und SymbolMittel Niedrig nderung Gelbton / Gelb-Wei-Kontrast bei Navigationslistenelement

Niedrig Niedrig Testen des Farbkonzepts bzgl. FehlsichtigkeitenNiedrig Mittel Bei Farbschema-nderung: Testing auf existente DesignrichtlinienMittel Mittel Reihung der Menpunkte bzw. Menhierarchie berdenken (Twitter vs. Nachrichten)Niedrig Niedrig Durchscrollen in der App ermglichenMittel Niedrig berschrift fr den Startpunkt der Navigation integrierenHoch Hoch Entscheidung berarbeitung Navigation oder Umbenennung der FunktionNiedrig Hoch Integration Liftausfall in NavigationNiedrig Niedrig Verstrkung der roten Umrandung bei ungelesenen NachrichtenNiedrig Mittel Andere Darstellungsform der Durchschnittsgeschwindigkeit, z.B. DurchschnittszeichenMittel Hoch Community-Funktion berdenken bzw. berarbeitenNiedrig Mittel Symbol im Hauptmen ndern, Httensymbol oder Messer&Gabel-Symbol verwendenMittel Niedrig Anzeige fr Liftfahrdauer als solche betiteln


17/42



For example, the one item with high priority was discussed in detail, because the term'navigation' was interpreted by many participants as describing the same functionality as the

navigation in cars. Lacking a better term it was decided to stick with the term 'navigation'.

3.4. Pilot 2

3.4.1. Setup

The focus for Pilot 2 were user experience aspects. To get feedback from as many participants aspossible we used self-administered questionnaires before and after the test run. So theparticipants had to fill out the questionnaires, from which we could analyse quantitative data.

Additionally, short interviews with all participants and one focus group (n=5) were conducted to

get more qualitative data. Pilot 2 was conducted on 5 consecutive days in which 54 participantstested the software during a timeframe of 2-5 hours. During this timeframe they had to fulfilltwo navigation tasks. Apart from the quantitative data measured via the logfile and ECC, there

will be a lot of qualitative feedback from the participants as well.

After a discussion about methodology the General Assembly in January in Graz, we decided tointegrate instant feedback. The participants had to give feedback (good / not good) about thequality of the displayed lift waiting time.

Figure 14: Instant feedback screen for lift waiting time

3.4.2. Results

As the participants had to fulfil two navigation tasks we got the most feedback about thisfeature. To fulfil the navigation tasks the participants had to go to a specific start point and selectthe target of the navigation. After that the route was calculated and displayed on the screen, see

Figure 15.


18/42



Figure 15: Route list in the navigation feature

Basically, the navigation feature was improved a lot compared to the version tested at Pilot 1.But still the participants missed some functionality like the usage of a completely free startingpoint (at the moment a specific POI must be used) and the re-calculation of the route in casesomeone drives in wrong direction. It can clearly be seen that such expectations come from the

fact that many people are already very much used to the navigation features of cars andsmartphones. The qualitative feedback of the participants showed that it was not clear to themthat a slope is by far not as structured as streets and that the movement on a slope is verydifferent to that on a street. A functionality which was evaluated as very positive was theautomatic display of a photo with a arrow in it to indicate the direction at an important point onthe slope, seeFigure 16.

In average each participant started 5 navigations, so the participants not only tried to fulfil thenavigation tasks but also used the navigation with their own start and target points. For 75 % of

these navigations the route could be loaded. This means, that in one of four cases the mobilenetwork connection was not available. Furthermore, from the loaded routes 49 % were finished,i.e. the selected target was reached. 39 % of the navigations were cancelled by the user and theremaining 12 % were not finished (e.g. because the test run ended before). The usefulness of thenavigation feature was rated with 3,2 (1 very useful, 5 not useful at all). This value underlines theneed for improvement if this feature should be further developed.

Figure 16: Automatic navigation hint with photo and arrow


19/42



Testing of lift waiting time difficult, as not so many people were on the slopes. This has to betaken into consideration when analysing the very positive value of the instant feedback for the

lift waiting time. 214 of 238 feedbacks (90%) confirmed a correct waiting time.

The sent notifications were considered to be useful and not too frequent. During the test runs

we sent 4 notifications in total (sent every 30 mins). The contents of the notifications were avoucher for a drink, a hint to use the instant feedback for lift waiting time, the announcement ofan event (Musikanten WM) and some advertisement for a hut. Especially the notifications abouthuts very interesting for the participants. 75 % of all send messages were read by the participants.

The usefulness of this feature was rated with 2,3 (1 very useful, 5 not useful at all).

The weather feature was again considered to be the feature which made the most impression of

being finished. Also the information about huts, their services and photos of the huts very founduseful by the participants. This feature was used in average 7 times per test run. The usefulnessof this feature was rated with 2,2 (1 very useful, 5 not useful at all).

The filtered Twitter stream was, like in Pilot 1, a feature not to be considered interesting. Onereason is the missing interactivity know from Twitter, as it was not possible to reply or forward atweet. This leads to the general topic of personalisation. Many participants wished to be able tohave more personalised content, e.g. posting from social networks like Facebook. This is a littlebit surprising as the focus groups and online survey showed a different picture. On the otherhand many participants had IT background and so have more affinity to social networks. Inaverage 12 tweets were received per test-run, 40 % of them were read by the participants. Theusefulness of this feature was rated with 3,9 (1 very useful, 5 not useful at all).

In summary Pilot 2 showed that the most qualitative feedbacks from participants can beclustered into three areas: Security aspects, technical problems, the smart factor. Regarding

the security aspects the major concerns are centred around the fact that moving your eyes awayfrom the slope to the display is a potential risk. Unfortunately, there were many differenttechnical problems which influenced the test result. This was especially a problem, whenparticipants very skiing together, because the problem of one goggle was transparent to the othertest persons as well.

The Bluetooth connection problems can be divided into a lost pairing between remote control

and the goggle or between the smartphone and the goggle. Some participants thenunintentionally triggered a factory reset which deleted our software. Furthermore, the GPS signalwas harder to catch on some devices than on others. As this was no problem in earlier tests andin Pilot 1, it seems a firmware update might be the reason for that. Another problem was thesometimes not working connection to the mobile network (this was limited and provoked bysome used smartphone models). Finally, the connectivity problems with the ECC were notdirectly visible to the participants, but the so induced stress at the test team also influenced theparticipants. The last of the three areas concerns the smart factor of the goggles. Obviously,many participants had higher expectations about the display (projected directly into the glasses)and the feature set (e.g. map-based navigation like in a car). A reason for that might be that many

of the participants had an IT background. Another reason might be that the media coverage of


20/42



the project was extensive and the software was thus more considered to be market-ready thanbeing a prototype (what it was).

Apart from the above, we have learned from a Quality of Service (QoS) perspective that theenergy consumption of our app the goggles is comparable to that of the original software. We

couldn't discover significant lower run times with Smart Ski Goggles. This means, it was noproblem to run the software 5 hours without any break. However, we have seen, that due to afirmware problem the recharging of the battery must be done with the software running,otherwise the battery is not being fully loaded. Additionally, in very cold conditions (belowminus 10 C) the battery life will be much shorter, it is recommended to use an additionalexternal battery back in this case. The average roundtrip time (time between user acquire data

from the backend and the requested data arrives at the client) was around 2 sec, whereas theacquisition of a navigation route data needed more time compared to all the other data, as therouting server had to calculate the route in real-time.

3.5.

Discussion

The approach to co-create software for smart glasses in a multi-step approach basically provedto be very useful. With integrating potential users from the beginning the features could beimplemented as close as possible to real needs and requirements. Of course, some of therequirements and wished features could not be implemented due to technical, conceptual orresource reasons.

On the other hand one could argue that integrating potential users in the conception of featuresfor a device which is so new that it barely hit the market is not a good idea. This has to be

confirmed if it comes to the fact that many people really believed the glass shield of the gogglewould be the display (like a HUD in a car). So, it is very important to make it clear to theparticipants of such a study how the display works. This can be achieved best by letting them trythe goggles by themselves.

Some delays in getting access to the turnstile data and the installation of the video cameras couldbe caught up until Pilot 2. Though, it was initially planned to have both finished for Pilot 1, sothe user feedback for the lift waiting time feature was not as valuable as it would have been whentesting under more crowded slope conditions during Pilot 1.

An important lesson learnt in this project is that, again, less is more. The integration of too manyfeatures forced us to reduce the detail level or bug fixing of a certain feature due to time andresource constraints. Especially in Pilot 2 it turned out that the testers were very keen to test thesoftware thoroughly. The user experience could have been increased with less functionality butbetter tested before.

A crucial point are system tests in which the complete system is being testing in real lifeconditions, as we have a complex system with many components (goggle, remote, smartphone,backend server, several other servers) with unpredictable real-time performance (mobile networkavailability, GPS availability and accuracy, etc) In a future project of this kind more focus has to

be put on system testing compared to feature development.


21/42



Another crucial point is the acquisition of suitable participants for testing and the studyguidelines. We have seen in Pilot 1 and especially Pilot 2 that the expectations of participants are

a major factor in their evaluation of the user experience of the device and the installed software.And if participants are testing together they are influencing each other which leads to a not soclear test result. So, the guidelines how to conduct such a study are of major importance as wellas the execution of the study itself.


22/42



4. Software Development

4.1. System Architecture

The Smart Ski Goggles system consists of three components. First, the Oakley Airwave is

defined as the frontend. It will be used to display real-time information, which is gathered fromthe integrated sensors and from the attached smartphone. The application running on the Oakley

Airwave is referred to as the Client Application.

The Bluetooth-connected smartphone runs the Smart Ski Goggles Gateway App. The GatewayApp is responsible for connecting to the EXPERIMEDIA components and exchange data fromthe client application to the Smart Ski Goggles Backend.

The Smart Ski Goggles backend processes information from number different external sources

such as weather service, resort information service, and navigation information provider. In

addition, it receives lift utilization statistics from external components like video cameras andembedded IR-beams sensors. The lift utilization statistics is analysed and used for providing liftcapacity utilization information to the clients. Furthermore, the Smart Ski Goggles backend alsoreceives turnstile usage information from each lift which is also used for capacity utilizationinformation for the clients. Finally, the Smart Ski Goggles backend will be used for pushingnotifications to the client. Examples will be urgent weather warnings, social updates like newtweets, hospitality offers, or unplanned lift open/close events.

The Smart Ski Goggles backend also provides an interface for administration. This is labelled asthe admin cockpit. In the admin cockpit, the administrators can configure send broadcastnotifications to all connected users. These notifications can be weather warnings, hospitalitynotifications, event notifications, and tweet notification. Unplanned lift open / close events areautomatically propagated to all connected clients.

How do they interact?

The Oakley Airwave is connected to the attached smartphone by Bluetooth. It uses a customdeveloped message protocol to request information and receive the related responses. Thegateway application also communicates with the EXPERIMEDIA components and the SmartSki Goggles Backend server over 3G cellular networks. The Smart Ski Goggles Backend server is

connected via high speed internet connection (100mbit/s) to the internet.


23/42



Figure 17: Overall system architecture

4.2. Client App

The Client App runs on the Oakley Airwave, in which the technological components aremanufactured by Recon Instruments. Oakley Airwave runs Android 4.1.2. Recon Instrumentsprovides a software development kit (SDK), which is used to access the integrated sensors andto receive skiing statistics. The Client App was developed in an agile approach throughout theproject and was available in v0.2 at Pilot 1, whereas v0.5 was used for Pilot 2. This is finalprototype version of the app.

Basically the user interaction is limited to the means the remote control of the data ski goggleoffers (buttons for up, down, left, right, enter, back). So, we implemented a menu-basednavigation. The horizontal menu of the top of the screen indicates by icons the availablefeatures. The currently used feature is being highlighted. With the "left" and "right" buttons theuser can switch between the features, whereas the "up" and "down" buttons are used to scrollthrough lists (e.g. lift waiting time) or switch between screens, in case a certain feature consists ofmore than one screen (e.g. weather). With the "enter" button a selected lift element can beopened, i.e. a detail screen appears.


24/42



The Client App was configured in a way that it loads automatically on boot-up of the device andcannot be exited. This was to ensure that the participants of Pilot 1 and 2 are really only using

Smart Ski Goggles and not accidentally the original software installed on the Oakley Airwave.

4.2.1. Features

Based on the general goal to provide real-time information to the user the following featureswere implemented:

Lift waiting time

Navigation to huts, lifts

Current weather and forecast

Notifications, e.g. warnings

Hospitality, e.g. special offers

Filtered Twitter stream

Individual info like speed and distance

The lift waiting time feature is implemented as a list, sorted in a way that the closest lift is on thetop position. The lift waiting time is indicated by coloured icons (green = no waiting time, yellow= short waiting time, orange = long waiting time), seeFigure 18.

Figure 18: Lift waiting time

The navigation feature was being improved substantially after the results from Pilot 1. The majorchallenge was to transform the result from the routing server, which was provided by an XMLlist, into a format which is displayable and understandable for the user. A huge problem here wasthat the routing information (from the routing server) itself many times was not correct. SomePOIs had wrong GPS coordinates or were wrongly connected, sometimes connections betweenPOIS were missing. For Pilot 2 we tried to improve the displayed routing information withadding arrow icons into the routing list, seeFigure 19.Furthermore, the usage of photographs

with arrows on it (seeFigure 20)helped users in Pilot 2 to identify an important point on theslope where he or she has to turn. These photographs were automatically shown in the display assoon as the user was as close as 50 meters to the displayed POI.


25/42



Figure 19: Routing list in the navigation feature

Figure 20: Automatic photographic hint in the navigation feature

A weather service (wetter.at) was integrated to provide current weather conditions as well as aforecast for the next hours and days, seeFigure 21 andFigure 22.As the screen space is limited

we only displayed the most important information (temperature, weather icon and wind speed).

Figure 21: Weather screen (current weather)


26/42



Figure 22: Weather screen (hourly forecast)

With the notification feature the service operator has got the possibility to push messages intothe Client App and inform skiers in real-time about important informations.Figure 23 shows anexample where a event is being announced. The feature is implemented in a way that anotification can contain a link to another feature or screen in the Client App. If, for example, aspecial offer for a hut is being sent, the message can contain a link to the hut's detail page in the

app.

Figure 23: Notification announcing an event

Within the hospitality feature all huts in the ski area are being shown in a list. After selecting acertain hut, there are several detail pages with text, photo (see Figure 24)and opening hours.Furthermore, it is possible to start a navigation to the hut in question.

Figure 24: One of a huts details screens

The feature "Individual Information" contains some basic data about the skier's activity, i.e.average and top speed (seeFigure 25), as well as the distance and height meters the skiers droveduring the last run. This feature was implemented as we knew from our first study that these

informations are of high interest to the users and were simply expected. Actually, theseinformations are already part of the original software of the Oakley Airwave.


27/42



Figure 25: Individual information (average and maximum speed)

An interesting featured was derived from the focus groups: If the speed is higher than 20 km/hthe screen shows only the speed, time and a notification icon (if a new notification has arrived),see Figure 26. This is to not distract the user. If a navigation is active only the routinginformation to the next POI is being displayed (instead of the speed).

Figure 26: Speed, time and notification icon, if user is in motion above 20 kph

4.2.2. Logging

For analysis purposes a debug mode and a logfile has been implemented. The debug modedisplays debugging information which is normally hidden from client. By default, the debugmode is deactivated.

The logfile is written at a regular interval and protocols the user activity as well as environmentproperties such as speed, GPS, battery level, and the related time stamp. Figure 27 shows an

extract of a logfile. The logfile is a comma-separated values (CSV) file format. Each entry startswith current date, current time, latitude, longitude, altitude, and followed by logging parametername and a variable number of log values for this parameter. For example, the fourth line islogging the "Weather-Start" event. It is triggered when a user enters the weather screen. Chapter4.5.1 covers the semantics of logfile in greater detail.


28/42



Figure 27: Small part of a logfile from a Pilot 1 testrun

4.3.

Gateway App

The Gateway App uses a network data caching system in order to counteract mobile networkoutages or network reception drops. In addition, it implements a prefetching mechanism so thatoften used data such as lift waiting time indication, and weather service are available in advance.Our caching approach proved to be very reliable. Only a navigational route can only becalculated if we have live access to the routing server. In future work this necessary live accesscould be avoided if we pre-fetch most used or even all possible navigation routes.

The Gateway App provides several configuration options, which affect the functionality. Forexample, it provides options to enable enhanced navigation mode (more information), altercache time out, skiing mode speed activation limit, or the geo-fence radius.

The Gateway App (on the smartphone) was incrementally improved based on the experience wehave during testing. In general the Bluetooth connection between Client App and Gateway App

was very stable and can easily be re-established if lost. However, during Pilot 2 a fewsmartphones had mobile network connectivity problems. This resulted in devices failing to fetchdata from the Smart Ski Goggles backend. Since the problems occurred randomly distributed, itassumed that it is due to a device specific problem.

4.4. Backend Server and CMS

Due to the fact that the installation of light beams in the area of the lift entrance is only possible

with huge efforts we concluded that we will only use the Skidata turnstile usage data and thevideo analysis from the AVCC component to indicate the waiting time at a lift.

The Smart Ski Goggles backend can be used for pushing notifications to the client. Exampleswill be urgent weather warnings, social updates like new tweets, hospitality offers, or unplannedlift open/close events. Unplanned lift open / close events are automatically propagated to allconnected clients. This feature is already finished, see Figure 28, and got very good feedbackduring Pilot 1. In addition, chalet owners are having the possibility to promote special mealoffers when there are any new or updated offers. This is being done manually in the Smart SkiGoggles backend (Admin Cockpit). Basic information about the chalet is per default integrated

into the Smart Ski Goggles software and can be administered via the backend.


29/42



Figure 28: Notification screen

The Smart Ski Goggles backend implements an ECC client. It is used for dispatching metricssuch as roundtrip time to external service, AVCC performance data, etc. The user interface ofthe ECC client is reduced as the ECC dashboard provides advanced facilities for monitoringsystem events.

The Smart Ski Goggles backend was developed using Panama, an open-source Lightweight Web-Application Framework. The backend uses a MySQL database for storing weather, lift, slopes,lift utilization and tweet information.

The Smart Ski Goggles backend provides a set of interfaces, which are exposed to the GatewayApp. The Smart Ski Goggles interfaces deliver data in a JSON data format. Since some externalservices (Feratel, wetter.at) deliver data in XML format, the data is being converted andoptimized for the Gateway App.

4.5.

Integration of EXPERIMEDIA ComponentsInitially it was also intended to use PCC for retrieving POIs. However, during inspection of PCC

it was revealed that the interface does not meet our requirements for saving POIs. In addition,the navigation provider also has a set of POIs. Due to technical reasons the navigation providercannot use external POIs. As a result the PCC component was not integrated in the experiment.However, the SCC (SAD service) and AVCC (VAS service) were used, as well as of course theECC.

4.5.1. Integration of ECC

The ECC should enable the near real-time monitoring of important experiment parameters (e.g.

status of 3G connectivity). Furthermore it was planned that provenance data will be reported tothe ECC, so it would have been possible to monitor in real-time at the ECC dashboard somedetermined actions the participants are doing during Pilot 2, e.g. starting a navigation or using aspecific lift. Unfortunately, the provenance feature was not finished by the baseline componentand thus could not be used in the experiment.

The following metrics were implemented and integrated into the ECC as well as written to thelogfile on the data ski goggles:

QoS


30/42



o Battery levels of Oakley Airwave and smartphone (Overall power consumption byapplication)

o 3G network reception level (Service availability)o GPS availabilityo Round trip time per feature call (Service performance measurement)o

Lift waiting time provided by AVCC

QoE

o Application Feature Call (To see which features are used where and when)o Application Feature Usage Duration (How long is a certain feature used?)o Application Feature Usage Count (How often were which feature used?)o Navigation Usage Details (Which starting points and targets were used? Was the

navigation cancelled or finished?)o GPS Position (To send location-dependent notifications)o

Time between reception and reading of notification (Analysis of usage behaviour)o Instant user feedback on quality of lift waiting time indication

In the preparation and during the Pilot 2 test runs a few problems with the dashboard appeared.The dashboard responded very slowly and it was difficult for the operators to monitor theexperiment. It was very hard to really monitor the experiment with ten client testing in parallel.

The reason was most probably the unstable mobile network connections of the clients to theECC RabbitMQ. Further investigation is required to determine the root cause of the problem.Figure 29 shows the number of measurement sets which were exported during Pilot 2 usingECC dashboard export function.

Figure 29: ECC measurement sets (the y axis shows their number)

During the usage of the ECC dashboard in Pilot 2, a number of issues came up:

0

50

100

150

200

250

300

350

31/03/2014 01/04/2014 02/04/2014 03/04/2014 04/04/2014


31/42



Each Smart Ski Goggles client reports 62 metric measurement sets. Since there were 11clients, there is theoretical limit of 682 measurement sets reported. However,Figure 29displays the number of actual measurement sets reported. It seems that not all availablemetrics were reported to the ECC dashboard based on the assumption that there was no

fundamental change of user behaviour. Changing a new user in the ECC dashboard was very slow. This might be due to the high

number of ECC metrics reported by each Smart Ski Goggles client.

On April, 1st, no clients appeared in the ECC dashboard. In order to fix this problem,the ECC dashboard had to be shut down and the RabbitMQ queues had to be deletedbefore starting a new dashboard.

One operator of the ECC dashboard reported that only few graphs showed actual data.However, this might be related to first issue reported earlier.

Sometimes an error happened and the dashboard became out of sync. As a result, thedashboard operator had to perform a browser refresh to fix this problem.

The ECC dashboard is an interesting technology to monitor events and user behaviour. Basedon our experiences above, we identified some recommendations for a future version of the ECCdashboard:

The responsiveness of the ECC dashboard should be improved. The time to select newusers or adding new graphs should be minimized in order to make the observation easier.

The stability of the ECC dashboard should be improved and sync issues should beavoided.

The dashboard should support multiple instances of the same experiment to make theobserving easier. This also provides the ability for multiple operators.

The dashboard should support observing measurements of the same type from multipleusers in one single graphs.

4.5.2. Integration of SCC

The SCC supported the experiment in providing a filtered stream of Twitter tweets with thefollowing filter criteria:

#Schladming or the word Schladming in the text body (no case sensitivity)

#Planai, @planai, or Planai in the text body (no case sensitivity) #Skiing

#Dachstein or Dachstein in the text body (no case sensitivity)

#Bluetomato

#SkiAmade

#SkiData

@Bluetomato

@SchladDachstein

@skiamade_info

#SSG2014


32/42



Here it is important that not too many tweets are pushed into the system. The SAD analysis wasconfigured to provide new tweets not more often than every hour. The Smart Ski Goggles

backend polled new tweets regularly and new tweets were automatically pushed to every singleconnected Gateway (and of course Client) App.

The SCC service worked very reliably (apart from single downtimes due to power outages of theserver infrastructure).


33/42



4.5.3. Integration of AVCC

The Smart Ski Goggles experiment was using a new AVCC component (VAS - Video AnalyticsService) from Joanneum Research (JRS) to provide information about lift usage by skiers. Thenecessary lift cameras were connected to a processing node provided by the Smart Ski Goggles

experiment, which ran the Video Analysis software. The processing node used the LAN networkhosted by the Schladming lift operator to connect to the Smart Ski Goggles backend. Thecameras were installed in the ski area especially for the experiment. Each VAS processing node(i.e. every lift) pushed a key performance indicator (percentage of waiting area covered bypeople) every 10 seconds to the Smart Ski Googles backend, where it was processed. Theintegration of the AVCC/VAS was easy and the performance and reliability was very good.Figure 30 shows how a lift camera is mounted to observe the waiting area at a lift entrance.

Figure 30: Lift camera


34/42



5. Business Model Generation and Exploitation

The business model in general describes which key resources are necessary to produce a certainvalue proposition (i.e. the product, service). Furthermore, it describes via which channels this

product or service is being offered and distributed to the customer. Finally, it opposes financialcosts to expected turnover.

For "Smart Ski Goggles" it was necessary to elaborate different alternative business model

scenarios, because there is not really a market in place so far for data ski goggles. So, it is ofspecial importance to evaluate different scenarios.

5.1.

Business model generation

5.1.1. Methods

The general basis for our approach is to use the Effectuation method combined with thebusiness model framework of Osterwalder/Pigneur, seeFigure 31Figure 31.This is based on thefact, that there are a lot of uncertainties with new technology like data ski goggles, as e.g. thetechnology acceptance of smart goggles at all is not very well researched.

Figure 31: Business Model Canvas (Osterwalder/Pigneur)

In general the scenarios we were working on provide alternative approaches from users buyingthe data ski goggles with "Smart Ski Goggles" software pre-installed to the ski resort operators

renting the data ski goggles with "Smart Ski Goggles" software on it to skiers.


35/42



To work out these scenarios of course some basic data was necessary. This was collected via theonline survey (see chapter3.2), from which we e.g. learned how much people are willing to pay

as a rental fee (in average roughly eleven Euro), see Figure 9. For the same purpose weconducted a series of interviews we have done with typical stakeholders of a service like "SmartSki Goggles". Thus, we gained insights into the needs and opinions of important partners in the

value chain. For a later exploitation this understanding is crucial.

Of course we also did some global online research to find out whether services like "Smart SkiGoggles" are available somewhere and how the business model there would work.

5.1.2. Results

In general there are two distinct approaches in distributing "Smart Ski Goggles" and thenecessary data ski goggles. The customers can buy the ski goggles or they can rent them. For thebuying option we found out that in general 69,4 % of the participants in the online survey are

basically willing to buy a smart ski goggle, see Figure 32.On the other hand, the average pricethese people were willing to pay was between 113,80 Euro (people who thought the gogglemustn't cost more than a regular ski goggle) and 160,70 Euro (people who thought the gogglecan cost more than a regular ski goggle). In any case, these price expectations are very much lessthan the current price of the Oakley Airwave (roughly 650 Euro). Of course, with higher

volumes and less brand-intensive goggle manufacturers the price will go down, but it is not likelythat it falls below 200 Euro in the coming years. So, for the buying models it has to be kept in

mind that only a very small target group can be reached.

Figure 32: Interest in buying a smart ski goggle (segmented in target groups)

5.1.2.1. Scenario 1 "Rental via Retail"

In this scenario the user has to rent the data ski goggles and the software "Smart Ski Googles" ispre-installed. With this approach more customers can be reached, but on the other hand the


36/42



rental logistics are more effort for the operator of the service. An important partner in thismodel is the sports retailer with the necessary experience for the rental process. An option would

still be that the rental process is being done by the ski resort operator himself - e.g. with renting adata ski goggle in the same step as buying the ski ticket.

Figure 33: Business model scenario "Rental via Retail"

5.1.2.2. Scenario 2 "Rental via Special Partner"

Scenario 2 is distinct from scenario 1 especially in the way how the data ski goggles are

distributed to the customer. Here hotels or VIP clubs are renting the goggles to their guests. Thiscan be either done for a fee (like in scenario 1) or as free premium service or as part of apackage. Of course, this approach makes it necessary that hotels or VIP clubs organise theirrental process by themselves.


37/42



Figure 34: Business model scenario "Rental via Special Partner"

5.1.2.3. Scenario 3 "Buying and Branding"

Scenario 3 is based on the approach that a branded version of a data ski google with the app"Smart Ski Goggles" on it is being offered to customers via regular retail or online shops.Depending on the marketing value which is being calculated by the ski resort this could evenlead to lower prices as if the customer would by the data ski goggle in an unbranded version. Theadvantage here is the simpler logistics process, but on the other hand the ski resort has to invest

in the data ski googles which at the end are potentially not being sold, which is a valid risk.

Figure 35: Business model scenario "Buying and Branding"


38/42



5.1.2.4. Scenario 4 "App Only"

The final scenario is based on the approach that only the app "Smart Ski Goggles" is being

distributed online (in the beginning potentially for free). This approach has the lowest risk forthe ski resport but also the potentially lowest marketing value.

Figure 36: Business model scenario "App Only"

5.2. Discussion

For all elaborated business model scenarios it has to be taken into account that probably none ofthese will lead to huge turnovers in the first years as the market is not that developed yet. But,apart from the turnover potential also the marketing potential has to be considered (see chapter6). "Smart Ski Goggles" is definitely a unique experience for guests and a ski resort candifferentiate itself from competitors through destination-specific information and services likee.g. real navigation. From a customer point of view for sure some improvements have to bedone to the software, e.g. personalization and implementation of new features. Of course, alsothe cost for reaching marketability have to be taken into consideration.


39/42



To discuss the elaborated business model scenarios with experts a business model workshop wasconducted in Schladming. With participants from the lift operator, the regions tourism office and

Schladming 2030 the different approaches were evaluated against feasibility and necessary pre-conditions to implement them in real life. A crucial discussion point was that the cost to furtherdevelop and test the current prototype of "Smart Ski Goggles" to get it market-ready has to keptlow, otherwise the ski resort wouldn't invest in a service with such a small volume of users.

Again, the marketing value as well as the indirect rentability (people reading about the service inthe media might be interest to come to the ski resort, even if they don't buy or rent the data skigoggle) have to be taken in account. Specific for Schladming was, that the media coverage aboutthe research project with "Smart Ski Goggles" was huge and so this marketing value was

according to the experts considered to be already exploited to a large extent.

In summary, the business model scenario 3 (Buying and Branding) was the one, which wasevaluated as most interesting of the four presented scenarios. Though, as it can be seen inFigure

37,in the end a combined approached was favoured. So, skiers should be able to buy a brandeddata ski google but it should also be possible to only rent it.

Figure 37: Best rated scenario at Business Model Workshop


40/42



6. Dissemination

We have issued a press announcement in November 2013 and got some very good press articlesin Austrian print and online media. During the Pilot 1 another press announcement was made by

our project partner Schladming-Dachstein, which resulted in many articles (more than 30!) in theAustrian and German press and online media, e.g.Figure 38.There was even a radio report inAustrias largest radio station (3), seeFigure 39,and one in TV (Puls 4).

Figure 38: One of the press articles about Smart Ski Goggles


41/42



Figure 39: Radio interview with test persons

Due to the fact that we have huge interest by the media we conducted a special test day formedia people. During this day the journalists could test "Smart Ski Goggles" by themselves andthey had the opportunity to ask detailed questions about the service.

The huge media coverage led to an extensive publicity of the project. This was good in the senseof making the research efforts in this EU funded project public. During Pilot 2 we learned thatthis publicity also had a disadvantage. Many of the participants already knew about the projectand the features of "Smart Ski Goggles". Obviously, they had the impression that the software isalready market-ready and some of them even thought the displayed information would be showndirectly on the goggles glasses. For the Pilot 2 this meant that these high expectations for sure

influenced the user's evaluation of the tested prototype.

The project will be submitted to two competitions in Austria: First, it will be submitted to the"Fast Forward Award" and later on to the "Steirischer Tourismusinnovationspreis".


42/42


7. Conclusion

In summary the experiment showed that in general there is a huge potential for using data skigoggles with apps like "Smart Ski Goggles". People are basically open to use data googles if they

offer an added-value to them. Though, security and privacy aspects have to be taken seriously.

The approach to follow a multi-step co-creation methodology proved to make sense, as theconcept could be developed in a very agile way, always close to the potential customers' needs

and concerns. Another important point was the inclusion of many stakeholders from thebeginning of the experiment to the very end. Only with these experts from different areas like skiresort operations, retail, tourism and marketing a holistic view of the service could be reached.

Technology-wise the operation of a service like "Smart Ski Goggles" is feasible, all technologiesneeded are in place and more or less conventional. The integration of available EXPERIMEDIA

components like e.g. the video analysis module (AVCC) made it during the experimentationeasier to implement certain functionalities. Though, strong support by local stakeholders isneeded (depending on type of data).

From a user's point of view it is crucial to have an easy and clear UI as well as an intuitiveinteraction concept. On top of that other user experience factors like the form factor, weight,battery life, price, etc. have to be taken into account if the service should be developed further tomarketability.

From a business perspective the difficulty is that the market for these devices has yet to be

developed. As long as there are no useful apps available many people won't buy such goggles.On the other hand as long as not so many devices are used no one wants to invest in developingsuch apps. This chicken-egg-problem is typical for the very early stage of new products and can

only be solved by brave entrepreneurs. This experiment have contributed to this in providingdeep insights into technological, organizational, economic and social aspects of such services.

In terms of exploitation the next steps should be to discuss the results of the experiment withinterested stakeholders (HW manufacturers, ski resorts, retailers, etc) and to further develop thebusiness model scenarios.

D4.10.3 SmartSkiGoggles Experiment Results and Evaluation v1.0

Documents

Transcript of D4.10.3 SmartSkiGoggles Experiment Results and Evaluation v1.0