ORIGINAL ARTICLE

A method to assess pervasive qualities in mobile games

Luis Valente1& Bruno Feijó2

& Julio Cesar Sampaio do Prado Leite3& Esteban Clua1

Received: 2 December 2016 /Accepted: 11 December 2017# Springer-Verlag London Ltd., part of Springer Nature 2017

AbstractSoftware designers are facing huge challenges imposed by a new generation of applications that mix real and digital worlds, suchas pervasive games. This type of game has recently become a worldwide phenomenon, with thousands of people walking in thestreets with smartphones to interact with the physical environment. In this paper, we propose a new method to assess pervasivequalities in pervasive mobile games, which can be customized and extended to other ubiquitous applications. This methodgenerates a quality report, which consists of a quality spreadsheet (containing metric values and comments) and a qualityvector (representing the game quality profile in the form of a bar chart). In addition, we can compare quality vectors usingsimilarity criteria. In this paper, we apply the proposed method to commercial and academic prototype games to shed more lighton their pervasive characteristics and identify ways to improve the overall quality that sets these games apart from traditionaldigital games—that is, pervasiveness.

Keywords Pervasivemobile games . Pervasiveness . Non-functional requirements . Ubiquitous systems . Game design

1 Introduction

Software designers are facing huge challenges imposed by anew generation of apps and systems that mix real and digitalworlds, mostly using natural interfaces. These systems arechanging the way we work, communicate, learn, and havefun. Examples of such new systems are intelligent personalassistants (e.g., Siri, Cortana, Google Now, and Echo), vir-tual reality systems, the internet of things, and pervasive

games. In a broad view, we can classify these apps andsystems as ubiquitous, in the sense that they can be oneverything, everywhere, and anytime, and can naturally in-tertwine real and digital worlds through multimodal inter-faces. Moreover, these systems present new characteristicsnot found in conventional software (such as being pervasive,immersive, affective, and enjoyable) and in some cases, theybring new paradigms of UI concepts [1]. These characteris-tics are quite distinct from the more concrete non-functionalrequirements in software engineering (e.g., Bsafety,^Breusability,^ and Befficiency^). They also contrast withthe more abstract attributes found in non-functional require-ments. They seem to be closer to intangible and non-measurable qualities, such as Bfun,^ Baesthetic,^ Bexciting,^and Bflow,^ which are subjective softgoals. For example,these kinds of requirements are common in digital games[2–4], deriving from emotional aspects regarding the gameexperience. Consequently, some researchers have referred tothese requirements as Bemotional requirements^ [2, 3],Baffective factors^ [5], and Bexperience requirements^[6]—this last one including gameplay factors, emotional fac-tors, and sensory factors. Therefore, we need to revisit theissue of quality requirements in software engineering forubiquitous apps, systems, and games.

In this paper, we approach this line of investigation byproposing a method to assess the Bpervasiveness quality^ in

* Luis Valentelvalente@ic.uff.br

Bruno Feijóbfeijo@inf.puc-rio.br

Julio Cesar Sampaio do Prado Leitejulio@inf.puc-rio.br

Esteban Cluaesteban@ic.uff.br

1 Institute of Computing, UFF, Av. Gal. Milton Tavares de Souza, s/n,São Domingos, Niterói 24210-346, RJ, Brazil

2 VisionLab/Department of Informatics, PUC-Rio, Rua Marquês deSão Vicente 225, Gávea, Rio de Janeiro 22451-900, RJ, Brazil

3 Department of Informatics, PUC-Rio, Rua Marquês de São Vicente225, Gávea, Rio de Janeiro 22451-900, RJ, Brazil

Personal and Ubiquitous Computinghttps://doi.org/10.1007/s00779-017-1107-0

pervasive mobile games. These games are a subset of perva-sive games where the main game playing device is mobile(e.g., networked, portable, and context-aware devices suchas smartphones and tablets). Examples of pervasive mobilegames are Pirates! [7], Botfighters [8], PAC-LAN [9],REXplorer [10], Zombies Run! [11], Spellbound [12],Ingress [13], and Pokémon Go [14].

Pervasiveness is the elusive quality that differentiatesBpervasive games^ from traditional digital games.Pervasiveness holds several meanings, such as being ubiqui-tous, permeating something, and spreading something (orsomewhere) in a physical space. In games (as in any otherapplication), we face enormous difficulties to define the roleof this elusive quality. In a previous work [15], we discusseddifferent (and often conflicting) approaches to the concept ofpervasiveness in games. Considering this scenario, instead oftrying to provide yet another, all-encompassing, theoretical def-inition of pervasiveness, we opted for a more practical approachby proposing a taxonomy of Bpervasive game qualities^—a setof quality attributes that contribute to characterize a game asBbeing pervasive.^ The driving question for the first version ofthis taxonomy was Bwhat makes a game pervasive?.^We iden-tified these qualities by analyzing 24 pervasive mobile games,reviewing the game literature, and using the authors’ experi-ences in developing mobile and context-aware games.

The main objective of our assessment process is to helpstakeholders [16] (e.g., interested parties, such as game de-signers, game developers, programmers, and researchers) inthe following two tasks: (1) creating new games that containthe desired pervasive features and (2) evaluating existing per-vasive mobile games, whether commercial or academic pro-totypes. Considering the first task, our assessment processoffers tools (e.g., Bquality vectors^ and Bchecklists,^ seeSection 2.5) to help developers, programmers, and designersto make informed decisions about which pervasive qualitiesthe game should support to create a product with better qual-ity. Concerning the second task, we foresee the followingbenefits:

a) Discover which pervasive qualities a game contains;b) Learn how a game applies pervasive qualities (e.g., design

and implementation), with the added benefit of possiblydiscovering design and/or implementations flaws that im-pair the pervasive quality;

c) Given a) and b), the stakeholder could detect issues thatmay hinder pervasiveness in an existing game, creatingthe opportunity to improve how that specific game appliespervasive qualities (in case the stakeholder is also thegame developer and/or designer);

d) Learn about pervasiveness in general by assessing severalgames, enabling the stakeholder to get acquainted withthis concept and even to extend our taxonomy later whenshe/he is more experienced;

e) Facilitate teaching, studying, and comprehending whatpervasive mobile games are and how they are structured;

f) Analyze several games to create (design and/or implemen-tation) catalogs and guidelines about how to apply specificpervasive features.

We believe that our assessment method can be adapted toother ubiquitous systems, although we have not yet validatedthis claim. In any case, tools, techniques, and guidelines toassess pervasive qualities are highly dependent on the specifictype of ubiquitous system. Therefore, we dedicate this paperto pervasive games on mobile platforms (i.e., smartphones,tablets, wearable devices), and we believe that the lessonslearned here may contribute to a better practice of pervasivequality assessment in the design of ubiquitous systems. Thereare different reasons for starting this type of investigation withpervasive mobile games. First, games are software applica-tions that face almost every aspect of computing challenges.Secondly, games permeate many different areas (from socialsimulation to playful learning). Thirdly, pervasive games haverecently emerged as apps with a strong (and sometimes unde-sirable) impact on large communities (e.g., Pokémon Go).

Some software attributes are not features of a system.Instead, these attributes are qualities (also called non-functional requirements); some examples are usability, reli-ability, maintainability, and portability. They are global con-straints on a software system. Software quality primarily con-cerns Bfitness to purpose^ and is highly dependent on theapplication domain [17]. This imposes a great challenge dur-ing design, because we need to predict how well the systemwill fit its purpose without having the opportunity to test it inthe future environments. This delayed feedback requires in-tensive use of models and simulations during the design stage.There is a vast literature on quality predictors and measure-ment methods for non-functional requirements [18–22].However, there are few studies about non-functional require-ments for ubiquitous systems and, especially, pervasivegames.

Taxonomies of pervasive qualities in games are rare (seeBRelated works^ in Section 4). In a previous work [15], weproposed a conceptual map of quality requirements that helpsunderstanding Bpervasiveness^ in pervasive mobile games(see Section 2.4 below). In this map, we decomposedBpervasiveness^ into two levels of non-functional require-ments, which we named Bpervasive qualities^ (Table 1).Additionally, this map contains a set of questions for eachquality requirement, which helps designers in evaluating orintroducing pervasiveness (and its subcomponents) in perva-sive mobile games. In this paper, we propose a method toassess pervasiveness in pervasive mobile games, which ap-plies the pervasive qualities and checklists found in [15].

We consider that assessing quality features in ubiquitoussystems requires a five-step macro process. Concerning

Pers Ubiquit Comput

pervasive mobile games, we explored the first four of thesesteps in [15]. In this paper, we enhance the fourth step andinclude a fifth step. We start by defining these steps for thecase of pervasive mobile games as follows:

1. Identify the fundamental quality. In the case of pervasivemobile games, this fundamental quality is Bpervasiveness.^Pervasiveness is the unique, subtle feature of pervasivesystems, while immersion is the key concept in virtual re-ality systems, for example. Other ubiquitous systems mayrequire more than one fundamental quality;

2. Classify the system by boundary criteria. Boundarycriteria help the requirements engineer to narrow the setof possible games to consider. For example, Table 2 pre-sents criteria we used to classify games as Bpervasivemobile games^ [15];

3. Decompose the fundamental quality into refinedrequirements. Table 1 presents a decomposition of perva-siveness into two levels of requirements [15]. At the firstlevel, Table 1 presents 7 non-functional requirements(qualities) that contribute to pervasiveness. On the secondlevel, there are 16 non-functional requirements that contrib-ute to the first-level qualities.We refer to this second level asBpervasive mobile game qualities,^ as they are qualitiesmore specific to pervasive mobile games. Using the lan-guage of the NFR framework [23], the first-level non-func-tional requirements and the pervasive mobile game qualitiescan be considered NFR softgoals. Table 1 describes thecontents of a NFR SIG (softgoal interdependency graph),and all hierarchical relationships in this table are equivalentto Bhelp^ contributions in the NFR framework;

4. Define checklists (questions) for each second-levelrequirement. In this step, we apply the checklists we pro-posed in [15]. However, in the present paper, we enhancethese checklists with the following elements: clarifyingcomments, a question classification system, and a set ofmetrics to be used in the assessment process;

5. Apply the assessment process. This is a six-step process,which is the main contribution in this paper. This process

aims to answer the list of questions associated with eachsecond-level quality requirement and to produce quantitativereports.

Figure 1 illustrates a simplified view of our assessment pro-cess. This assessment process allows the requirements engineer togenerate a final quality report that summarizes the quality assess-ment of a particular pervasivemobile game after analyzing a largecollection of information about this specific game. The qualityreport consists of a quality spreadsheet (containing metric valuesand comments) and a quality vector (representing the qualityprofile of the game in the form of a bar chart). Also, qualityvectors can be used to measure quality similarities between dif-ferent games. When developing this assessment process, the au-thors produced several supporting materials (e.g., mark-ups, key-words, and indications of information groups) that are useful forsoftware engineers and designers. Indeed, practitioners can usethis material to highlight important information and produce rel-evant annotated material related to first-level qualities, whichleads to a detailed analysis of second-level qualities.

This paper is organized as follows. Section 2 presents theassessment process we propose in this paper. Section 3 pre-sents a summary of two case studies. Section 4 discusses re-lated works. Section 5 presents some final conclusions anddirections for future studies.

2 Assessing pervasive qualities in pervasivemobile games

Figure 2 illustrates a BPMN diagram1 (Business ProcessModel and Notation) that includes our six-step process toevaluate pervasive game qualities (in the Bevaluator^ lane).This diagram features four actors: taxonomist, author,

1 The reader should note that we annotated the BPMN (i.e., the traced line andthe indication of hierarchy) to better explain the levels in the assessment pro-cess. We chose to avoid using a swimlane in this picture (i.e., the actor), asthere is only one actor in this case (the user of the process).

Table 1 Pervasive game qualitiesthat contribute to pervasiveness[15]

First-level qualities Second-level qualities

Spatiality (Spa) Mobility (Mob), Local space redefinition (LSR)

Permanence (Perm) Persistency (Per), Daily life interleaving (DLI)

Communicability(Comm)

Connectivity (Con)

Accessibility (Acc) Access usability (Usa), Device independence (DI), Transmediality (TM), Gameautonomy (GA)

Context-awareness(CA)

Game content adaptability (GCA), Game object tangibility (GOT)

Resilience (Res) Uncertainty handling policy (UHP), Game pacing (GP)

Sociality (Soc) Social communication (SC), Conformance to physical and social settings (CPS),Involving non-players (INP)

Pers Ubiquit Comput

evaluator, and end user. Currently, the Btaxonomist^ actor re-fers to the present authors. Next, the Bauthor^ actor representsall the authors who create and publish information sourcesabout the game, including the game application itself.

Sections 2 to 2.5 describe each step the Bevaluator^ actorconducts to assess pervasive game qualities. The Bevaluator^may produce a Breport^ at the end of the process, which couldbe accessed by Bend users^ interested in learning about per-vasiveness aspects in a given game. The Bend user^ actorrepresents game designers, developers, programmers, re-searchers, and requirement engineers.

2.1 Searching for available information sources (Steps1 and 2)

The process starts by selecting an information source. Accordingto [24], information sources are not restricted to people (alsoreferred as stakeholders, users, or clients) but may include differ-ent sorts of documents, such as books, internal memos,

specifications, scientific literature, news, product descriptions,and manuals. Identification of information sources is fundamen-tally important to elicit information about the focus of interest(pervasive mobile games in our assessment process).

We have used different information sources to elicit pervasivegame qualities including research literature and media assess-ments (e.g., informal materials, reports, non-peer-reviewed pa-pers, web sites, interviews, presentations). Sometimes, media as-sessments are the only available information source about a game(e.g., commercial games that are no longer available). These as-sessments are usually rich, full of player and expert user knowl-edge about the gameworld, and as such, they are valuable input tounderstand how a game works. Also, many pervasive mobilegames are research prototypes unavailable to the general public,but they usually have academic papers published elsewhere.Finally, there are commercial games available in applicationstores, allowing developers, researchers, and end users to down-load and play them. For our purposes, we keep all these informa-tion sources in a database. We identify the following types ofinformation sources that we can use in our assessment process:

IS1. (Bpapers^) Academic papers;IS2. (Bofficial site^) The official game website, which may

contain information about the main game features (i.e.,commercial, selling points);

IS3. (Bvideos^) Videos describing the game. These videosmay be official (e.g., created by the game developersand companies) or not (e.g., videos featuring gameplaymade by players);

Fig. 1 Simplified view of our assessment process applied to a game. We analyze highlighting information related to first-level qualities and then weanalyze the corresponding second-level qualities using the checklists. The quality vector summarizes the final report. Some parts are computer-assisted

Table 2 The criteria to classify games as Bpervasive mobile games^ in[15]. The asterisk denotes a mandatory condition

Condition Description

C1* Games that use context information

C2* Games that use mobile devices as the main game interfaces

C3 Games that access remote data on the move

C4 Multiplayer games

Pers Ubiquit Comput

IS4. (Bnews^) Game news websites, which may contain in-formal non-academic articles (e.g., blog posts, reports)and game reviews. These (informal) reviews may bewritten by players and mainly depict their impressionsabout the gameplay;

IS5. (Bdev articles^) Informal, non-academic articlesthat designers/developers of a game write aboutthat specific game. In the game industry, thereare Bpost-mortem^ articles that describe detailsabout the overall game development process,which may include information about what wentright and wrong about the game design, imple-mentation, and deployment. These articles mayappear in websites dedicated to game developers;

IS6. (Binterviews^) Interviews (e.g., text, video, and audio)with the game developers about the game;

IS7. (Bpresentations^) Presentations and talks in game de-veloper conferences;

IS8. (Bhands-on^) Hands-on experience, which includes de-veloping games and evaluating games developed byother people. In the latter case, it means downloadingand playing the game to conduct a custom evaluationprocess;

IS9. (Bmarketing^) Marketing campaigns about the game.

2.2 Determining whether a game is a pervasivemobile game (Step 3)

The borderline of pervasive and non-pervasive games may befuzzy. We consider a game to be a Bpervasive mobile game^if:

& The game uses context information in game activities (i.e.,the condition C1 in Table 2). For example, the game mayuse context (e.g., location) to create game content.Another possibility is to modify and adapt game rulesaccording to contextual information;

& The game implements at least one game interface incontext-aware mobile devices (i.e., the condition C2in Table 2). Examples of context-aware mobile de-vices are smartphones and tablets. We consider agame a Bborderline pervasive game^ if the game istransmedia (see [15] for a discussion on Bpervasivegames and transmediality^).

Fig. 2 The complete BPMN process diagram describing our process to assess pervasive game qualities

Pers Ubiquit Comput

The first task to classify a game as a pervasive mobile gameis to explore an adequate information source by searching forspecific keywords and expressions (mark-ups) that are typicalof pervasive games (mobile or not). The following tables canbe used for this task: Table 3 lists common names (for gamegenres and game styles) that refer to pervasive games; Table 4lists keywords and expressions that refer to context-awaremobile devices that pervasive games may use (which is partof condition C2 in Table 2); and Table 5 lists general keywordsand expressions related to pervasive games. These mark-upscome from analyzes of a large number of pervasive mobilegames [15, 25] we conducted in other works.

Additionally, the following tables in Section 2.4 (i.e., Step5—identifying first-level qualities) can be useful to assist inrecognizing a pervasive mobile game: Table 6 (Spatiality,which also strongly relates to condition C1) and Table 10(Context-awareness, which relates directly to condition C1).Computer programs could assist this part of the assessmentprocess by highlighting the abovementioned keywords andexpressions (mark-ups) for visual and auditory inspections.Also, the following questions may aid in determining whethera game meets the C1 and C2 conditions of Table 2:

& Which devices do players use to play the game?& Are these devices portable and context-aware?& Which sensors does the game use?& What kinds of context information does the game use in

game activities?

We claim, based on our experience, that if the informationsource is an academic paper (IS1), we should first browse thepaper title and abstract, searching for keywords and expres-sions that match those in Tables 4, 5, 6, and 10. This type ofinspection of academic papers is usually sufficient to decidewhether a game is a pervasive mobile game or not.

If all the abovementioned procedures are not effective toclassify the game as a mobile pervasive one, we can use theprocedures of the next step (Step 4, Section 2.3), which iden-tifies useful groups of information that are helpful to assess thefirst and second levels of pervasive qualities. We claim, alsobased on our experience, that if the available informationsource is non-structured or informal (e.g., IS2 to IS7IS7—Bofficial sit,^ Bvideos,^ Bnews,^ Bdev articles,^Binterviews,^ Bpresentations^), we should briefly explore it

(concerning Step 4) before deciding whether the game is apervasive mobile game or not.

2.3 Identifying useful information groups (Step 4)

We can classify useful types of information to assess pervasivequalities into the following groups:

IG1. The Bdesign^ group, which may be found in informa-tion sources under the following names: Bgame design,game description, game plot^. This information groupprovides descriptions about the overall game design,main game features, game narrative, game story, gamemechanics, and game activities. This informationgroup usually appears in information sources IS1 toIS7 (i.e., Bpaper,^ Bofficial site,^ Bvideos,^ Bnews,^Bdev articles,^ Binterviews,^ Bpresentations^);

IG2. The Bimplementation^ group, which may be recog-nized in information sources by the following names:Bimplementation, game architecture, game infra-structure.^ This information group provides implemen-tation details, system architectures, game components,and related topics. This kind of information is useful toassess pervasive qualities that have aspects influencedby implementation issues, such as Uncertainty han-dling policy, Connectivity, Persistency, and Game ob-ject tangibility. This information group usually appearsin information sources IS1, IS5, IS6, and IS7 (Bpaper,^Bdev articles,^ Binterviews,^ and Bpresentations^);

IG3. The Bevaluation^ group, which may appear asBdeployment, experiments, validation, user tests, eval-uation, discussion, post-mortem.^ The evaluation pro-cess may be conducted by researchers with actualplayers or with the developers themselves, constitutingformal or semi-formal processes. This kind of infor-mation is useful to assess aspects that are better ob-served while playing the game. Pervasive qualities thatinclude these aspects are Uncertainty handling policy,Game pacing, Social communication, Conformance tophysical and social settings, Connectivity, Game con-tent adaptability, Game object tangibility, and Localspace redefinition. This information group usually ap-pears in information sources IS1, IS5, IS6, IS7, andIS8 (Bpapers,^ Bdev articles,^ Binterviews,^ andBhands-on^);

Table 3 Common genre/stylenames of pervasive games Transreality game Exergame Transmedia game Urban storytelling game

Mixed-reality game Live-action game Mobile mixed-reality game Augmented reality game

Location-based game Ubiquitous game Crossmedia game Ambient game

Urban game Tourism game Outdoor game Computer-augmented game

Context-aware game

Pers Ubiquit Comput

IG4. The Bgameplay^ group, which may appear as Bgameplaythrough, walkthrough, hands-on.^ This informa-tion group describes the gameplay dynamics. The maindifference from the IG3 group is that in IG4 there is noevaluation (i.e., it is an informal process). Informationsources that usually contain this group are IS3, IS4, andIS8 (i.e., Bvideos,^ Bnews,^ and Bhands-on^).

If the information source is textual, we may browse it tohighlight these groups in the text (if possible). The IG1 andIG2 groups can be useful to determine whether a game is apervasive mobile game if the previous step did not provideenough information to classify the game.

If the information source is not text-based (e.g., audio, vid-eo), we may need to browse it more carefully, taking notes toidentify these information groups. Although an informationsource may have these information groups, their existencedoes not guarantee that there will be enough information toassess the pervasive game qualities.

Some types of information sources may not contain allinformation groups—for example, a hands-on evaluation pro-cess (i.e., IS8 information source) will not provide implemen-tation details to assess the Uncertainty handling policy quality(Section 2.4.6) for a given game (assuming the Bevaluator^does not correspond to the game developer). In this case, thisassessment may require other types of information source,such as IS1, IS5, IS6, and IS7 (i.e., Bpapers,^ Bdev articles,^Binterviews,^ and Bpresentations^).

2.4 First-level quality analysis (Step 5)

In Step 5, we browse the information groups (if any) identifiedin the previous step to conduct a preliminary analysis,highlighting and taking notes of some keywords, expressions,and excerpts related to the first-level qualities. The objectiveshere are twofold. The first is to get acquainted with the

information source, especially concerning the quality attri-butes. Secondly, we want to elicit the most relevant first-level qualities of the game. This analysis may help to conductStep 6—assessing second-level qualities using the checklists.

This section presents common keywords and expressions(mark-ups related to first-level qualities) that help inconducting Step 5. These keywords and expressions do notrepresent all possibilities. These mark-ups come from our pre-vious works, based on the analysis of a large number of per-vasive mobile games [15, 25]. This section briefly describesthe first- and second-level qualities. For detailed descriptions,please refer to [15].

2.4.1 Spatiality

Spatiality refers to aspects such as the physical place wherethe game happens and mobility. The physical place where thegame happens (e.g., Bthe game scene,^ Bgame area,^ Bgamearena^) usually corresponds to outdoor and public areas ofurban places, but these places may also be dedicated indoorphysical installations customized for a specific game.Spatiality is composed of Mobility and Local spaceredefinition.

Mobility (Mob) refers to two aspects. The first is mobilecomputing—using wireless connections on mobile devices.The second corresponds to the requirements of movement inthe game and the physical size of the game area, which mayrequire players to move through great distances to completegame activities.

Local space redefinition (LSR) refers to how the game isable to change the meaning of the places where the gamehappens, for example, by augmenting value to physical places(using virtual content), integrating the physical place in thegame (or objects in this place), using actors to portray non-playing characters, helping the player to learn about the phys-ical environment, and having the players perceive the

Table 4 Expressions andkeywords related to context-aware mobile devices

Mobile platforms (e.g., iOS, Android) Specific mobile device manufacturers Smartphone Tablet

Wearable devices Mobile phone

Table 5 Common keywords andexpressions related to pervasivegames

Augmented reality Liveperformance

Interactions with physicalobjects

Blending physical and virtualworlds

Computer-augmentedgame

Smart objects Pervasive platforms Brings game to the physical world

Exploration anddiscovery

Pervasiveness Players act out in thephysical world

Combination of virtual and realworlds

Extend the virtualworld

Play in real life Beyond the device screen Ubiquitous computing gamingenvironment

Ubiquitous computing Play in publicplaces

Pers Ubiquit Comput

environment with alternative viewpoints. These ideas are dif-ferent from just Bbeing in a place^ and using some physicalproperty (e.g., location) without explicit reference to the localsurrounding context.

A preliminary assessment of Spatiality consists of iden-tifying information that describes how the game usesphysical space and its elements. For example, some relatedterms are Bplay area,^ Bgame area,^ Bplay space,^Bplayground,^ and Bgame space.^ The physical spacewhere the game happens may be a public place or aprivate (and customized) place. This information may helpto assess the second-level qualities (Local space redefini-tion and Mobility) using the checklists. Another usefulinformation relates to specific locations and physical ob-jects (e.g., statues, landmarks) the game uses. This infor-mation may help to assess Local space redefinition,Mobility, Game autonomy (Accessibility), and Game con-tent adaptability (Context-awareness). Table 6 lists com-mon keywords and expressions related to Spatiality.

2.4.2 Permanence

In an ultimate state of pervasiveness, a pervasive mobile gamebecomes a game world that exists as an independent entityparallel to Breal life.^ The most prominent aspect of this ideais the possibility to engage with a process that evolves contin-uously over time (and does not depend on whether the playeris connected to the game or not). Permanence denotes the ideaof a game world being a persistent entity available anytime,anywhere. Permanence is composed of Persistency and Dailylife interleaving.

Persistency (Per) refers to maintaining the game state overtime for future game sessions over time. This quality is a

requirement for games that aim at simulating a parallel worldthat evolves by itself. This includes all games that have socialnetworking or community aspects. However, not all pervasivegames require persistency.

Daily life interleaving (DLI) refers to how a pervasive mo-bile game is able to diffuse through the daily lives of players,interleaving game playing with non-game activities. Forexample, many traditional games for mobile phones areBcasual games,^ meaning they have simple gameplayand players use them as Bintervals of daily life.^ Thisquality includes aspects such as persistent game worlds,short-play sessions, and asynchronous communication ofgame-related events to players.

While assessing Permanence, we search for clues, key-words, and expressions that describe game session details—for example, (1) How long a game session lasts, if completinggame activities require multiple game sessions, and (2) Ifgame sessions are independent (e.g., they do not depend onprevious game sessions).

We also search for information about persistent virtualworlds and how the game communicates with players whenthey are not engaged in the game. Persistent worlds in perva-sive mobile games may evolve or degrade independently ofplayer input or participation. Table 7 lists common keywordsand expressions to search for Permanence.

2.4.3 Communicability

Communicability concerns aspects related to how thegame communicates with players (and vice-versa).Communicability contains one second-level quality,which is Connectivity (Con). This quality refers to net-working usage in pervasive mobile games. For example,an important aspect is the connectivity scope that agame requires—global, local, or both. Global scope re-fers to using networking solutions as 3G networks.Local scope refers to using Wi-Fi and Bluetooth tech-nologies. Global scope enables the game to connect re-mote peers and to use networking in wide-range areas.Local scope enables the game to connect co-locatedpeers (e.g., players and servers). The size of the physi-cal space where the game takes place directly influencesconnectivity requirements.

Assessing Communicability consists of identifying thecommunication structure that the game uses. For example,the game may be single-player (no connectivity), single-player connected, and multiplayer. Observing the communi-cation scope (co-located, global, or both) is also important.The information group IG2 (i.e., Bimplementation^ group) isuseful to learn details about Connectivity, in the next step.Table 8 lists common keywords and expressions to searchfor Communicability.

Table 6 Common keywords and expressions related to Spatiality

Annotate thephysicalenvironment

Collect virtualcontent in thephysical world

Computationalelements in thephysical world

Explore aphysical place

Useinformationabout thephysicalspace

Landmarks Locality Location-based

Place content inphysicallocations

Physicalmovement asgamemechanic

Physicalnavigation

Physical placeaugmentedwith virtualcontent

Physical worldas gameboard

Play withlocation

QR codes Spatialexpansion

Walk in theenvironment

Wide-area game Outdoor game Physical world asstage

Play in publicplaces

Mobility

Pers Ubiquit Comput

2.4.4 Accessibility

Accessibility refers to how players can access the game andissues that might prevent a player from accessing the game,including social, economic, and technological issues.Accessibility is composed of Transmediality, Game autono-my, Access usability, and Device independence.

Transmediality (TM) refers to providing different game in-terfaces (i.e., modes of participation or game roles) using dif-ferent types of devices to deliver a broader transmedia expe-rience. These devices might be context-aware mobile devices(such as smartphones), desktop computers, web-based inter-faces, wearable devices, environment devices, or custom hard-ware [15]. In transmedia pervasive games, each game inter-face is associated with a type of device, enabling players toplay in different roles in the game. These roles may be collab-orative or not. Epidemic Menace [26] is an interesting exam-ple of a pervasive game that uses this quality.

Game autonomy (GA) refers to the degree to which a gameis able to work as an Bindependent system^—requiring nospecific environment setup or live game management (i.e.,orchestration). In games with high game autonomy (i.e.,Bautonomous games^), players are able to readily play thegame anywhere, anytime. Games with low game autonomyrequire a specific place, often with dedicated infrastructure(e.g., sensors spread in the environment, dedicated Wi-Fi net-works, public displays) to support the game activities. Also,games with low game autonomy may be considered as Beventgames^ because these games have a scheduled time to startand last for a specific duration.

Access usability (Usa) corresponds to traditional usabilityissues from human-computer interaction focused on mobiledevices and accessibility. Device independence refers to thepossibility of using a game interface in multiple platforms(e.g., Android, iOS, and web).

To assess Accessibility, we browse the information groupsidentified in the previous step looking for information andclues that describe issues that might affect access to thegame—considering social, economic, and technological

aspects. Table 9 lists common keywords and expressions forAccessibility. Besides, other examples of useful clues and in-formation are the following:

1. Concerns about including several groups of people ingame activities. For example, integrating people with dis-abilities in game activities;

2. Concerns about providing access to technology that wasnot widely available. For example, lending devices toplayers;

3. Concerns about using devices that are already common tothe general public in game activities. For example, design-ing games that use current smartphones and tablets toimplement the main game interfaces;

4. Concerns about defining a target user group and providingamenities to reach this group. For example, designinggames for children.

2.4.5 Context-awareness

Context-awareness helps create the mixed-reality that perva-sive mobile games provide. In particular, this quality refers tousing acquired and context information in game activities.This quality is composed of Game content adaptability andGame object tangibility.

Game content adaptability (GCA) relates to pervasive mo-bile games that generate game content dynamically using sen-sor and context data. This quality concerns how the game isable to keep the designed functionalities anywhere regardlessof context availability conditions.

Game object tangibility (GOT) relates to how the gameuses mobile devices as Btangible objects^—an object thatplayers grasp and manipulate in game activities, which mayhave a specific purpose (e.g., a role) in the game narrative.

We start assessing Context-awareness by searching forgeneral uses of context in game activities. Common types ofcontext in pervasive mobile games are location (e.g., GPS,

Table 8 Common keywords andexpressions related toCommunicability

3G/4G networks Ad hoc networks Asynchronous communication Asynchronous gameplay

Bluetooth Cellular connection Chat Client-server

Co-located play Multiplayer Networking NFC

SMS Text messaging Wi-Fi Mobility

Communities Social network

Table 7 Common keywords andexpressions related toPermanence

Embed play with daily life Interweaving with everyday life MMORPG Persistent world

Pervasive RPG Play anywhere, anytime Short-play sessions Temporal expansion

Temporality

Pers Ubiquit Comput

specific physical objects, QR codes), spatial area information(cell-id, Wi-Fi), proximity (e.g., beacons, QR codes, RFIDtags, Bluetooth, NFC). Other possibilities are physiologicalinformation (e.g., heart rate), interaction context (gestures,embodied interactions), traveled distance, speed, weather in-formation, social context, social network profiles, and envi-ronment information (e.g., noise level, temperature, light lev-el). Table 10 lists common keywords and expressions tosearch for Context-awareness.

2.4.6 Resilience

Resilience refers to the capacity of a game to keep thegame experience smooth despite technological issues thatmight break or disrupt this experience while players areengaging with the game. For example, technology compo-nents in pervasive mobile games (especially sensors andnetworking) have inherent limitations that generateBuncertainties.^ Uncertainties arise from Bseams,^ whichBarkhuus et al. [27] defined as Ba break, gap or ‘loss intranslation’ in a number of tools or media, designed foruse together as a uniformly and unproblematically experi-enced whole.^ Seams appear due to some of the followingfactors: limited or variable availability, limited or variablecoverage, limited accuracy and precision, intermittent op-eration, sudden malfunctions, and other issues. Resilienceis composed of Uncertainty handling policy and Gamepacing.

Uncertainty handling policy (UHP) refers to how a gamehandles seams and uncertainties in game activities, which gen-erally takes form through five general strategies: remove, hide,exploit, reveal, and manage [28–30].

Game pacing (GP) refers to two aspects: (1) how the paceof game activities may disrupt the operation of technologycomponents and (2) how the operation of technology compo-nents may limit the pacing of game activities. Usually, thesetechnology components correspond to sensors and network-ing infrastructure.

In assessing Resilience, we search for general uses ofsensor information and networking, which are the mainsources of uncertainties in pervasive mobile games. Inparticular, we pay attention to problems and issues re-ported in the information source and also try to foreseepossible issues not anticipated by the game developers.Table 11 lists common keywords and expressions tosearch for Resilience.

2.4.7 Sociality

Sociality refers to the game’s social aspects, being composedof Social communication, Conformance to physical and socialsettings, and Involving non-players.

Social communication (SC) refers to how game activ-ities enable players to communicate and socialize. Thegame may enable social communication between distrib-uted and/or co-located players. In the first case, thegame mediates this process using technology artifacts,such as mobile devices and network infrastructure. Inthe second case, players may communicate face-to-facedirectly (without technology artifacts) or by using thegame as a mediator (through technology).

Conformance to physical and social settings (CPS)refers to issues such as ethics, privacy concerns,conforming to social conventions, social context, safetyconcerns (e.g., protecting players from harm while mov-ing around), and adequacy to physical settings (e.g.,using audio feedback in noisy places may not workwell).

Involving non-players (INP) refers to integrating non-players into the game. Montola et al. [31] identified five par-ticipation modes for non-players in general pervasive games:BThe outsiders can be obstacles (Cases A and D), witnesses(Cases C, F, and J), an audience (Cases G andM), or tokens tobe collected (Case I) and they can be invited to become fullyfledged player–participants (Cases B, E, and J).^ In this paper,we add four possibilities to this list: Bhuman actors asperformers,^ Baudience as co-participators,^ Bbeingapproached by players^ (a general case where non-playersinteract with players but do not become a player), andBsources of game content^ (a general case that include Btokensto be collected,^ by Montola et al. [31]). Notably, applyingthis quality in games might raise ethical issues.

Assessing Sociality means searching for clues, key-words, and expressions that convey how the game isable to influence and involve people through game ac-tivities—these people may be players and non-players.Table 12 lists common keywords and expressions tosearch for Sociality.

2.5 Applying checklists to assess second-levelqualities (Step 6)

In Step 5, we produced annotated material that highlights in-formation about first-level qualities of a game. In Step 6, we

Table 9 Common keywords andexpressions related toAccessibility

Crossmedia Current smartphones Event-based game Haptics feedback

Multimodal user interface Public display Voice interaction Transmedia

Multi-platform Game prop Orchestration Game moderators

Pers Ubiquit Comput

browse this material to apply the checklists [15] of eachsecond-level quality. In this paper, we extend the materialwe described on our previous work [15] with metric rulesand additional comments. To calculate quality measures, wecreated a spreadsheet, which we name as Bqualityspreadsheet^ in this paper.2 This spreadsheet implements themetric rules (detailed in Section 2.5.1) and calculates scoring(i.e., metric values) automatically as the evaluator (e.g., de-signer, requirements engineer) answers the questions.

2.5.1 Quality vector and metrics

Each question in the checklist of a quality attribute is designedto reveal a particular aspect of this attribute. We can have aquantitative assessment of these quality attributes by calculat-ing a special vector called the Bquality vector.^ The qualityvector V is an array of values vi such that vi ∈[0,1], i = 1, n,where n is the number of second-level qualities (n = 16, in thispaper), and the value vi is the Bsupport level^ of the quality i,given by the sum of the support levels of each question asso-ciated with the quality i. For example, if the qualityBPersistence^ has three questions and only two answers fullysupport the aspects stated by these questions, then the supportlevel of this quality is 2/3.

Filling the quality vector is always an imperfect processbecause the following perfect conditions can never beachieved: (1) a complete set of reliable information sources,(2) a complete and accurate set of questions (i.e., a perfectchecklist), and (3) flawless human analysts. If conditions areperfect, the value viwould be an accurate measure of quality i.Therefore, in practice, we have only a proxy of each qualityattribute. Nevertheless, this is not a matter of great concern,because the idea behind the quality vector V is not to establishan accurate way to measure quality but to propose an instru-ment to assist the evaluation process of quality attributes dur-ing the design of ubiquitous apps and games.

There is another important issue affecting quality vectors.Awell-known fact in requirements engineering is that qualityattributes are interdependent and cannot be achieved in isola-tion. In a previous work [15], we started analyzing the inter-dependences between pervasive quality attributes, and weproposed some basic relationships (i R j for Bi is required byj,^ i + j for Bi helps j,^ and i − j for Bi hinders j,^ where i and jare two different quality attributes) that can be affected bysymbols of modality (e.g., a for Balways holds^ and m forBmay hold, in some specific cases^). Examples in [15] areUHP aR GCA (i.e., Uncertainty handling policy is alwaysrequired by Game content adaptability), Usa – Mob (i.e.,Access usability hinders Mobility), and GP m– TM (i.e.,Game pacing may hinder Transmediality in some cases).Therefore, no app or game can have an Ball-ones qualityvector^ (i.e., a vector with all elements equal to 1.0), becausesome quality attributes may hinder others (e.g., Accessusability hinders Mobility). We could use these interdepen-dence relationships as a consistency check for a given qualityvector, but we think we need further investigation into thismatter. Nevertheless, we can always use quality vectors as atool to help us identify these relationships.

We can calculate vi as a simple arithmetic mean, consider-ing that 0.0 means an Bunsupported^ aspect of the qualityattribute and 1.0 means a fully Bsupported^ aspect. Usually,the answer Bno^ corresponds to the value 0.0 and Byes^ to 1.0.Considering the checklists we elaborated in [15], this simplecalculation is not always possible for a number of reasons:

& Some questions are eliminative, in the sense that if theiranswers deny support to a particular attribute, then vi willbe 0.0 regardless of the other answers. However, in the

2 The companion website (http://www.ic.uff.br/~medialab/papers/2017/pauc)contains a report with all checklists and additional comments. This websitealso provides the quality spreadsheet for public use.

Table 10 Common keywordsand expressions related toContext-awareness

Beacons Bluetooth Body sensors Cell-id Cellularpositioning

Context Context-aware computing Context-dependentgameplay

Embeddedinteractions

Embodiedinteractions

Gesture-basedinteraction

GPS Implicitinteractions

Locationaware

Locationsensing

Location-based NFC Physiological input Proximity QR codes

RFID tags Change rules based oncontext information

Sensor networks Sensor-basedinput

Sensor-basedinteraction

Sensors Social context Tangible objects Wi-Fi

Table 11 Common keywords and expressions related to Resilience

Deal withuncertainty

Exploreseams

Orchestration Seamfuldesign

Seams Uncertainties Gamemoderators

Pers Ubiquit Comput

case of eliminative questions, if an answer gives support toan attribute, then it can participate or not in the calculation;

& Some answers are not simply Byes^ (1.0) or Bno^ (0.0),that is, they can be an expression, a list, or a number amida range of values;

& Sometimes the answer Bno^ means a supported aspect(1.0) and a Byes^ means an unsupported aspect (0.0);

& The support level should consider more than one informa-tion source.

For these reasons, we expand the simple calculation men-tioned above towards a more general concept, which we callBpervasive quality metrics^ in this paper. These metrics com-prise a set of equations, a set of hypotheses, and a set of rules.The simplest metric has the following equation:

vi ¼ αi1

Ni∑Ni

j¼1aggregate 1qij;2q

ij;⋯; Tq

ijÞ i ¼ 1; n

�ð1Þ

In Eq. 1, where kqij means we are browsing the information

source k to assess the second-level quality i, looking for theanswer to question j, this answer represents the support-levelvalue. There may be T information sources available (i.e., k =1...T). Currently, there are 16 second-level qualities (i.e., i =1...16). Finally, j varies according to the number of questionsfor each second-level quality.Ni is the total number of answersto be used in the calculation of vi; αi is 0.0 if there is at leastone eliminative answer and 1.0 otherwise; n is the total num-ber of second-level qualities (n = 16 in this paper); andaggregate is the aggregate function, which combines the con-tributions of each information source. The simplest aggregatefunction is the max function, which calculates the maximumof a set of values. Table 13 presents a case where the qualitynumber is 5 (i = 5), with 3 questions (e.g., a, b, c), and thedatabase has 2 information sources (T = 2). The aggregatefunction in this example is max.

Then, vi = (0.8 + 1.0 + 0.3) / 3 = 0.7. The simplest set ofhypotheses is the following:

1. All information sources in the database are equallyreliable;

2. The most important source is the one that gives thehighest level of support (therefore, the most adequate ag-gregate function ismax). An alternative hypothesis is Bthemost important source is the one that is more restrictive,i.e., the one that gives the lowest level of support.^ In this

latter case, the adequate aggregate function is min, whichcalculates the minimum of a set of values. Another pos-sibility is to assign degrees of importance for each infor-mation source (in this case, several aggregate functionscan be used, such as a weighted mean function, the maxfunction applied to the support level multiplied by aweight);

3. Each question in the checklist of a quality attribute revealsa different, and independent, aspect of this attribute (there-fore, the max or min functions can be interpreted as aunion operation, which collects support evidence from anumber of different information sources);

4. All the answers are equally reliable;5. All the questions for a specific quality attribute have the

same importance (therefore, the simple mean of Eq. 1 isan adequate formula). If this hypothesis does not apply,then we should change Eq. 1 into a weighted mean, i.e., aweight wj should multiply the aggregate function with thefollowing restriction: w1 +w2 +…=Ni;

6. The eliminative nature of a question is independent of theinformation sources (therefore, the parameter αi of Eq. 1is unique for each quality attribute i);

7. The database has complete information, that is, theBclosed world assumption^ applies to questions (i.e., ifan answer cannot be extract from an information source,then the aspect is considered unsupported).

The ways of combining answers (extracted from distinctinformation sources) are more diverse than those we proposeabove. Thus, many other metrics can be defined. In any case,we should always reason in terms of the set of hypotheses. Wehave already noted that different aggregate functions (e.g.,max, min) represent different hypotheses. However, there aremore radical ways of defining aggregation. For example, ifhypothesis 3 does not apply (i.e., the questions do not

Table 13 Example of answers for two information sources (shadedcells mean Bunknown support^)

Answers

Quality i Question j Info source 1 Info source 2

5 a 0.8 0.5

b 1.0

c 0.3

Table 12 Common keywordsand expressions related toSociality

Actors Approach passersby Approach strangers Bystanders

Chat Challenge social rules Invite spectators Non-players

Online participants Performers Social expansion Social interaction

Social interaction in public places Social networks Communities Mobile community

Spectators Audience User-generated content

Pers Ubiquit Comput

correspond to different aspects), then we may consider aggre-gating values at the information source level (instead of thequestion level). For example, in Table 13, we first calculate thesimple mean in each information source column and thenapply the max function on the set of these mean values (i.e.,v5 = max((0.8 + 1.0 + 0.0) / 3, (0.5 + 0.0 + 0.3) / 3) =max(0.60,0.27) = 0.60). In this case, the set of equations tocalculate vi is the following:

kSi¼ αi1Ni

∑j¼1

Ni

kqij i ¼ 1; n; k ¼ 1; T ð2aÞ

vi ¼ aggregate 1Si2Si ;⋯TSiÞ;� ð2bÞ

where kSi is the total support level for quality i obtained fromthe k-th information source (where T is the total number ofinformation sources in the database) and the other elementsare the same variables found in Eq. 1. The alternatives pro-posed for hypotheses 2 and 5 may also apply (e.g., aggregateas a weighted mean function).

As far as aggregation is concerned, the differencebetween Eq. 1 and Eq. 2 is that the first is an aggre-gation at the question level and the latter is an aggre-gation at the level of information source. In our experi-ence, Eq. 1 produces better results (especially becausewe always try to guarantee hypothesis 3). However,more experimental work is necessary to give moresound guidelines about types of aggregation.

We can apply several other metrics. Our fundamental rec-ommendation is twofold: ensure that the hypotheses are clearand search for simple metrics—even when a metric soundsoversimplified, do not give up the principle Bthe simpler, thebetter.^ This attitude is in line with the well-known Ockham’srazor principle [32], which can be restated here as Bamongcompeting hypotheses, the one with the fewest and simplestassumptions is the best.^ In any case, we should always re-member that there are many other sources of error and noisethat we cannot control, such as the humans involved in theprocess. In other words, there is no gain in searching for so-phisticated metrics when considering these issues.

We implement the abovementioned metrics as a qualityspreadsheet,3 which calculates the quality vector automatical-ly as the designer or requirements engineer answers the ques-tions. Figure 3 illustrates this spreadsheet, in which the inputcells are marked in blue and the quality vector is in the lastcolumn. This spreadsheet implements two different metricsusing the max function: one uses Eq. 1 and the other usesEq. 2b. Some questions can be assessed more adequatelythrough the IG3 information group (i.e., Bevaluation^ group).We highlight these questions in the spreadsheet by markingthem with an asterisk (e.g., the UHP1* symbol in the first rowin Fig. 3).

A helpful way to present the quality vector is through a barchart, which we interpret as a kind of quality spectrum or

Fig. 3 Quality assessment spreadsheet sample (incomplete). The input cells are marked in blue and the quality vector vi is in the last column

3 Available at http://www.ic.uff.br/~medialab/papers/2017/pauc

Pers Ubiquit Comput

quality profile (see Fig. 4 for an example). This bar chartpermits a quick global visualization of the quality attributesof a specific app or game. We may be tempted to use thesecharts (i.e., quality vectors) to compare two games or evaluatehow far a particular game is from an ideal quality profile. Thecomparison of two different games may make some sense, aswe explain below, but the concept of an ideal, prototypical, orabsolute quality profile is nonsensical. Moreover, the compar-isonwith an all-ones quality vector is wrong, because a qualityvector with all elements equal to 1.0 cannot exist (as we ex-plained at the beginning of this section).

The two quality vectors in Fig. 4 are visually quite similar(almost the same bars and values). Here, we start exploring theadvantages of comparing two quality vectors, and later, wepresent methods to calculate similarity values (which we con-sider as falling between 0 and 1). If the similarity is equal to 1,we will have a situation in which the quality attributes areequally taken into consideration, as far as the informationsources are concerned (i.e., we will have the same supportlevels). There are two basic situations in which we can checkquality vectors: one is for the same game (using differentinformation sources) and the other is for different games. Ahigh similarity value means that the quality vectors share sim-ilar quality concerns. However, in the case of comparing twodifferent games, this high value of similarity does not meanthat the two games are of the same type or provide a similargame playing experience.

The main advantage of comparing two quality profiles is toassist designers and requirements engineers during the processof evaluation and/or design. The assessment method we pro-pose in this paper is supposed to be customized by the user,who should decide on which aggregate function is more ade-quate and how to use the various sources of information. Thequality vectors may work as an indicator, like a dashboardelement. Although the precise definition of similarity and thecorrelation coefficient are given in the next paragraphs, thefollowing examples may help the reader understand the utilityof the quality vectors:

& We can compare the vector qualities that come from twodifferent information sources for the same app/game. Ifthe information sources are complete (in the sense of pro-viding information that can answer all questions), the vec-tor qualities should be similar. Thus, if the similarity is toolow, the analysts should consider the possibility of facingerrors or inadequate information sources;

& Whenever we have a pair of different games with similar-ity values of 0.5 or greater, it is worth investigating wheth-er one of the games can inspire new features for the otherone. This is a valid practice because they share similarquality concerns. For instance, we applied our method4

to the games Pac Map [33] and Exploding Places [34]and found a similarity value equal to 0.43 (Figure 5). Infact, they have a similar quality profile, but, for example,Exploding Places misses CPS (Conformance to physicaland social settings). This suggests that we could improveExploding Places in some Sociality aspects by answeringtwo important questions: BDo game activities possiblydisturb non-players?^ (CPS2) and BDo game activitiesexpose players (or non-players) to embarrassingsituations?^ (CPS3). However, we should always considerwhether a missing quality (or a quality with a low supportlevel) is a consequence of a poor information source or afaulty design;

& If a specific app or game has a Bgood reputation^ amongusers,5 then we can use it as a reference and check how faranother game is from this reference in terms of qualityattributes. In this case, we should always be aware ofhow complete and reliable the information sources are.Sometimes, a low similarity value is a consequence ofusing a bad information source;

Fig. 4 Similar quality vectors for hypothetical games X and Y

4 These assessments are available at http://www.ic.uff.br/~medialab/papers/2017/pauc5 Here, we assume that if players generally consider a given pervasive game asBgood^ (i.e., satisfying, enjoyable), then this game should have a Bgood^pervasive experience. However, we are aware of how difficult it is to explainwhy players would consider a game Bgood.^

Pers Ubiquit Comput

& Similarity values can be a quick way to explore a largedatabase of quality vectors, either to select a similar vectorgiven a specific one or to extract interesting pairs of gamesfor a more detailed analysis.

We can compare the similarity of two quality vectors usinga measure of similarity based on standard distances (e.g.,Eqs. 3 and 4) or using correlation methods (e.g., Pearson’scorrelation coefficient, Eq. 5). In this paper, we call the firsttype of measure BEuclidean similarity^ and the latter BPearsonsimilarity.^

d x; yð Þ ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi∑i

xi−yið Þ2r

ð3Þ

Esim x; yð Þ ¼ 1

1þ d x; yð Þ ð4Þ

r x; yð Þ ¼∑ixiyi−

∑ixið Þ− ∑iyið Þnffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

∑ix2i −

∑ixið Þ2n

!∑iy

2i −

∑iyið Þ2n

!vuutð5Þ

We use the Euclidean distance when we want to compareabsolute values (i.e., in case where the quality vectors aresupposed to be nearly identical). This may be the case whenwe are comparing the quality vectors from two different setsof information for the same game and we are expecting similaroutcomes.Whenwewant to compare the shape of the Bqualityspectrum^ regardless of their absolute values, we should usePearson’s correlation coefficient. For example, Fig. 4 illus-trates two quality vectors with low similarity by Euclideandistance (Esim = 0.56) and a high correlation coefficient(r = 0.86) for two different (hypothetical) games. However,we can use both measures of similarity (Euclidean orPearson) for the same game (but from different informationsources) or for different games—the choice depends on whatwe are looking for. For instance, we can use Euclidean dis-tance to compare two different games if we suspect that theirquality attributes are equally taken into consideration.

Finally, in the metrics, we have a set of rules (Bqualitymetric rules^) to calculate αi and kq

ij for each quality i. The

rules are the same for all types of information sources.Table 14 describes these rules and the types of questions.Table 15 presents a generic sample scale to assess otherBtext&scale^ and Btext&scale,REV^ questions. Table 16 pre-sents a sample scale to use when evaluating Btext&scale^questions related to technology uncertainties (e.g., seeUncertainty handling policy, Mobility, and Connectivity in[15]).

3 Case studies

This section presents the summarized assessment (onlyAccessibility and Sociality due to space constraints)6 of twopervasive mobile games, Spellbound [12] and Ingress [13].We used the quality spreadsheet with Eq. 1 to calculate thequality vectors (Fig. 6). The similarity values (Euclidean =0.37 and Pearson = 0.29) are low (as expected) because thesegames are quite different and the quality attributes are notequally taken into consideration.

In this section, we use the quality symbols (see Tables 17and 18) to indicate the sij values of each question of a qualityi—e.g., Access usability (Usa1 = 1, Usa2 = 1) for Accessusability in Spellbound. In this example, the answer to ques-tionUsa1 (BDo game activities require players to focus on thedevice screen all the time?^) is Bno^ (which produces thevalue 1, because the question type is Byes/no,REV^). Theanswer to question Usa2 (BDo game activities use variousmodalities to interact with the player?^) is Byes^ (which pro-duces the value 1). In addition, for the sake of uniformity, weuse B–^ in case of Bhint^ questions, as these questions have nosupport value, e.g., Transmediality (TM1 = 0, TM2 = 0,TM3 = 0, TM4 = −, TM5 = −). The reader should consultTables 17 and 18 to follow the analysis below and to under-stand the sij values.

6 For the complete analysis, please refer to http://www.ic.uff.br/~medialab/papers/2017/pauc

Fig. 5 Quality vectors for PacMap and Exploding Places, with Euclidean similarity = 0.43

Pers Ubiquit Comput

3.1 Spellbound

Spellbound is an outdoor pervasive mobile game that takesplace in a preselected game area, where two teams compete inthree quests (game activities). In these quests, players are ableto interact through proximity with virtual elements, whichhave corresponding real locations in the game area. The gameis designed to foster face-to-face communication by applyingthe Bheads up display^ paradigm and not using digital com-munication channels. Spellbound uses whole-body interac-tions (e.g., jumping, spinning, running) as the main interactionmechanic with virtual elements.

We begin the assessment process by searching for suitableinformation sources about Spellbound. We found a journal

paper [12] (an IS1 information source) and used it (sec-ond step) to determine if this game is a pervasive mo-bile game (third step). Browsing the paper title andabstract reveals keywords and expressions related topervasive games, such as Bmobile games beyond thedevice screen,^ Bimmersive game experiences integratedwith the real world,^ Boutdoor pervasive team-basedphysical game,^ Breal-world actions like jumping andspinning,^ Bvoice interaction,^ Bglanceable and hapticfeedback,^ and Banchors enjoyment on physical action,social interaction, and tangible feedback.^ This informa-tion seems enough to classify Spellbound as a pervasivemobile game, but we proceed to the next step for con-firmation—the highlighted expressions refer to mobile

Table 14 Question types andtheir descriptions (where d/ameans Bdoes not apply^ and n/ameans Bnot available^)

Question type Rules

yes/no The answers can be either Byes^ or Bno,^ where Byes^ means kqij = 1.0 and Bno^ means

kqij = 0.0 (e.g., Mob1 in Mobility—BDoes the game activities need to use networking

while the players are moving while playing?^). Answers Bd/a^ and Bn/a^ mean kqij = 0.0.

yes/no,REV: The answers can be either Byes^ or Bno^, where Byes^ means kqij = 0.0 and Bno^ means

kqij = 1.0 (e.g., GA1 in Game autonomy—BIs the game bound to specific places or local

context?^). Answers Bd/a^ and Bn/a^ mean kqij = 0.0.

text A textual answer, where any valid answer means kqij = 1.0. If the question does not apply, this

means that kqij = 0.0. For example, in BWhat are the connectivity requirements for the

game?^ (e.g., Con1 in Connectivity), valid answers could be Blocal networking^ orBglobal networking.^ In case there is no networking (e.g., it does not apply), the valuewould be kq

ij = 0.0. Answer Bn/a^ means kq

ij = 0.0.

text&scale A textual answer and a value for kqij (between 0.0 and 1.0) that represents the answer’s

adequacy according to a Likert scale (e.g., Tables 15 and 16)—the Bmost adequate/^answer means kq

ij = 1.0 and the Bleast adequate^ means kq

ij = 0.0. The Bmost adequate^

answer means that the aspect occurs as the Bhighest value/strongest intensity^ possible.For example, BHow does the game handle player privacy?^ (CPS1 in Conformance tophysical and social settings) may have the answer: Bthe game identifies players bynicknames^ and kq

ij = 0.5 (Badequate, ok^ in Table 15). Another example is BHow does

the game handle technology limitations?^ (e.g., UHP2 in Uncertainty handling policy),which may have the following answer: Bthe game identifies all uncertainties, but handlessome^ and kq

ij = 0.8 (Table 16). Answers Bd/a^ and Bn/a^ mean kq

ij = 0.0.

text&scale,REV A textual answer and a value for kqij (between 0.0 and 1.0) that represents the answer’s

adequacy as a Likert scale (e.g., Table 15)—the Bmost adequate/^ answer means kqij = 1.0

and the Bleast adequate^ means kqij = 0.0. Contrary to the previous type, here, the Bmost

adequate^ answer means that the aspect occurs as the Blowest value/weakest intensity.^For example, BDoes the game require players to walk long distances?^ (Mob4 inMobility)may have the answer: Byes, they need to walk 3 km^ (which might be considered Bhigh^)and kq

ij = 0.25 (Table 15). Answers Bd/a^ and Bn/a^ mean kq

ij = 0.0.

…,ELI A suffix for any of the types above to represent Beliminative^ questions. This means that ifthe answer to this question is kq

ij = 0.0 (e.g., Bno^ for a Byes/no,ELI^ question), then the

corresponding pervasive quality is considered Bunsupported^ regardless of the answers toother questions that the given quality might have. For example, if the answer to theBtext,ELI^ question BWhat are the connectivity requirements for the game?^ (e.g., Con1 inConnectivity) is Bno networking^ (or Bd/a^—does not apply), then kq

ij is calculated as 0.0

(because there is no network) and the quality BConnectivity^ is totally eliminated (i.e., αi

is set to 0.0).

hint These questions do not contribute to the quality metric calculation. Their purpose is toprovide useful information to assess other questions and the overall quality. For example,the TM3 question of Transmediality (BDoes the game balance the game experience for thevarious game roles?^) helps to identify a case where transmediality might not be supported(e.g., in case its answer is Byes^). On the other hand, answering Bno^ to TM3 does notbring insight to assess the overall quality.

Pers Ubiquit Comput

devices and integration of virtual and real worlds. In thefourth step, we browse the paper [12] to find specificinformation groups:

& IG1 (Bgame design^): Section 1, 3, and 4;& IG2 (Bimplementation^): Section 3;& IG3 (Bevaluation^): Sections 5 and 6.

Sections 3 and 5.1 (in [12]) describe the devices thatplayers use in this game: an Android smartphone, wearabledevices (wristband and two armbands)—two kinds of mobilecontext-aware devices (i.e., the C2 condition)—andearphones. Section 1 also describes context information thatthis game uses—location (i.e., the C1 condition).We concludethat Spellbound is a pervasive mobile game according to C1and C2 conditions in Table 2.

3.1.1 Accessibility

Spellbound is an interesting case of Access usability (Usa1 =1, Usa2 = 1), as this game was designed to avoid havingplayers look at the smartphone screen constantly by applyingBheads up play^ (Usa1). Moreover, the player is able to useseveral modalities (e.g., voice input, haptics feedback, andembodied interactions such as jumps, spins, and runs) to in-teract with the game (Usa2). Therefore, Access usabilityreaches 1.0 in the graph in Fig. 6 (i.e., Eq. 1 Vusa = 1.0).

Concerning Device independence (DI1 = 0), the game in-terface in Spellbound is built using Android devices and

custom hardware, which means there is no device indepen-dence (DI1).

Regarding Game autonomy (GA1 = 0, GA2 = 1, GA3 = 1,GA4 = 0), this game is bound to a specific place—apreselected playing field (GA1). However, this game doesnot require preparing the physical space for game sessions(GA2). Instead, artists and designers map virtual content tophysical locations using software tools. The game detects thevirtual content locations using GPS. The game does not re-quire orchestration while game sessions are happening (GA3).Finally, the game requires custom hardware—the armbandand wristband were designed solely for Spellbound.

Considering Transmediality (TM1= 0, TM2= 0, TM3= 0,TM4=−, TM5 =−), in this game, players compete in differ-ent teams (roles) using the same game interface (TM1, TM2).Because TM1 and TM2 are eliminative questions,Transmediality is unsupported and question TM3 does notapply. This game interface is composed of a smartphone, awristband, and two armbands (one for each arm). The gameuses the smartphone to collect sensor information to detectand process whole-body interactions. The players also usethe smartphone to send voice input to the game and to receiveaudio feedback. The wristband contains LEDs and vibrationmotors that the game uses to convey game state information toplayers. The game uses the armbands to assist players whenthey search for virtual content in the physical place, byinstructing them to turn right or left. Concerning questionTM4, there are no support modules for players (TM4).Finally, Spellbound is not part of a broader transmedia expe-rience (TM5).

3.1.2 Sociality

Considering Social communication (SC1 = 1, SC2 = 0,SC3 = 1, SC4 = 0, SC5 = 0), Spellbound provides an interest-ing case of a game design that deliberately avoids using digitalcommunication channels as a means to improve and fosterface-to-face communication in co-located situations (SC1).In this regard, Spellbound uses embodied interactions to en-courage heads up play and uses team play in a small area toaccomplish this goal. By using heads up play, players are able

Table 15 Sample scale forgeneric Btext&scale^ andBtext&scale,REV^ questions (asmentioned in Table 14)

Assessment Grading(text&scale)

Grading(text&scale,REV)

Very high, very strong, very good, all, very severe, mostadequate

1.0 0.0

High, strong, good, quite a bit, severe 0.75 0.25

Average, fair, ok, moderate, adequate 0.5 0.5

Low, weak, poor, mild 0.25 0.75

None, least adequate 0.0 1.0

Does not apply, not available 0.0 0.0

Table 16 Classification of Uncertainty handling policy consideringuncertainty identification and applied policies (as mentioned in Table 14)

Conditions Grading

Identifies all uncertainties, handles all 1.0

Identifies all uncertainties, handles some 0.8

Identifies some uncertainties, handles some 0.5

Identifies none, handles some 0.4

Identifies all uncertainties, handles none 0.25

Identifies none, handles none 0

Pers Ubiquit Comput

to pay attention to the surrounding environment and engage inconversations with teammates directly.

Concerning SC2, Spellbound does not transform the rela-tionships among players. However, according to the gameevaluation [12], it seems that team play in Spellbound mayhave provided socialization opportunities for players, as thefollowing excerpt describes: BSome players said doing theactions alone outdoors would make them feel self-consciousbut doing them with their team made them feel like they wereBpart of a secret club that others had no idea about^ and thatmade playing more fun^ [12]. This is a clear way to improvethis pervasive quality in Spellbound. Concerning SC3, thisgame requires players to approach passersby and interact withthem in the Recruit task. Regarding SC4, the game does notuse technology to create communities around the game.

Lastly, it seems that the game does not present emergentgameplay (SC5).

Regarding Conformance to physical and social settings(CPS1 = 0, CPS2 = 0, CPS3 = 0.75, CPS4 = 0, CPS5 =0.75), the information source [12] does not provide enoughinformation about privacy issues in Spellbound (CPS1). Also,because this game takes place in a preselected playgroundarea, it seems that there are no issues regarding local socialconventions and etiquette (CPS4). However, the Recruit gametask requires players to approach passersby, which might dis-turb non-players (CPS2). Also, the embodied interactions re-quired to play this game (e.g., jumps, spins, and runs) might bean issue for non-players (CPS2), although Sra and Schmandt[12] did not report that. We have here a clear identification ofquality issues that can be improved.

Table 17 Checklists of second-level qualities for Sociality. Theasterisk denotes questions thatcan be that can be answered moreadequately through evaluationactivities (e.g., hands-on evalua-tion, user tests, and experiments)

Social Communication SC1* (text&scale) How does the game use technology as means to improvecommunication among people?

SC2* (text&scale) How does the game transform the relationships amongplayers?

SC3 (yes/no) Does the game stimulate players to approach/start interactionswith other people?

SC4 (yes/no) Does the game use technology to foster communities or socialnetworks?

SC5* (yes/no) Does the game foster emergent gameplay? (yes/no)

Conformance to physical andsocial settings

CPS1 (text&scale) How does the game handle player privacy?

CPS2* (text&scale,REV) Do game activities possibly disturb non-players?

CPS3* (text&scale,REV) Do game activities expose players (or non-players)to embarrassing situations?

CPS4* (text&scale) Do game activities conform to local social conventionsand etiquette?

CPS5 (text&scale) Are game activities adequate to the physical setting of thegame?

Involving non-players INP1 (text.ELI) Does the game involve non-players? How does it do it?

INP2 (yes/no) Does the game have activities where players need to find outwho the other players are?

INP3 (yes/no) Does the game generate/use content that is based on other(non-player) people?

INP4 (yes/no) Does the game use human actors for non-player characters?

Fig. 6 Quality vectors for Spellbound and Ingress (using Eq. 1). Similarity values are Euclidean = 0.36 and Pearson = 0.27

Pers Ubiquit Comput

Concerning CPS3, Sra and Schmandt [12] reported thatsome players felt uncomfortable performing the embodiedinteractions when alone but not when they were in groups.Some players also felt uncomfortable when using certain re-quired keywords in voice input (such as Bhootie^) in theBFind^ game task, as passersby would be able to hear it.

Voice input is an important interaction mechanism inSpellbound. This game takes place in outdoor places, whereambient noise is likely to be high. The game developers in-cluded some measures to adapt this interaction mechanism tothe physical setting where this game takes place (CPS5). Thefollowing excerpt illustrates this issue: BIn outdoor spaceswith high ambient noise, there is always a possibility for theplayer to not hear the spoken dialog or remember what wasspoken so the system needs to be designed such that repetitionis possible on demand^ [12].

Finally, when evaluating Involving non-players (INP1 = 1,INP2 = 0, INP3 = 0, INP4 = 0), the answer to question INP1is Byes,^ because players need to approach passersby in theRecruit task. However, Spellbound does not have activitieswhere players need to find out who the other players are(INP2). Also, this game does not generate content based onnon-players (INP3), and Spellbound does not use actors norperformers (INP4).

3.2 Ingress

Ingress is a multiplayer mobile location game where players(the Bagents^) take part in two opposing teams (BEnlightened^and BResistance^). The game creates a mixed-reality worldbased on maps of real locations around the players. This gamepopulates this mixed-reality environment with Bportals,^which are virtual objects created according to the location of

real landmarks and objects such as monuments, historicalsites, statues, stores, churches, train stations, and public art.An important player goal is to capture these portals and claimthem for their team—an action that requires players to go thephysical location that corresponds to the portal. Players areable to link portals located in different areas to create Bcontrolfields.^ The control fields are important, as they are able togenerate a significant amount of XM (Bexotic matter^), whichis a kind of energy particle players need to collect while theywander around. Some player actions, such as interacting withportals (i.e., Bhacking portals^ in the game’s idiom), requirevariable amounts of XM that depend on the portal status.These portal linksmay span varying distances, from a hundredmeters to several kilometers (i.e., their length depends on thedistance between two portals).

Players interact with the game through the BIngressscanner^ (i.e., the smartphone). Besides interacting with por-tals, this scanner enables players to perform other actions,such as chatting with other members of their team, managinggame item inventory, and receiving news about live gameevents (e.g., a player of the opposite team is attacking a portalheld by the player) that happen locally, regionally (e.g., city-and state-wide), or worldwide.

We begin the assessment process by searching for suitableinformation sources about Ingress. There are several informa-tion sources about this game (e.g., official websites, videos,interviews, academic and non-academic papers), including thegame application itself, which is available for download in theGoogle and Apple application stores. Browsing the officialwebsite (https://www.ingress.com/) is enough to understandthat Ingress is a pervasive mobile game, as it runs oncontext-aware smartphones (the C2 condition) and it relieson context (i.e., location) information (the C1 condition).

Table 18 Checklists of second-level qualities for Accessibility.The asterisk denotes questionsthat can be that can be answeredmore adequately through evalua-tion activities (e.g., hands-onevaluation, user tests, andexperiments)

Access usability Usa1* (yes/no,REV) Do game activities require players to focus on the device screen all thetime?

Usa2 (yes/no) Do game activities use various modalities to interact with the player?

Deviceindependence

DI1 (yes/no) Is it possible to use the game interface across multiple mobile deviceplatforms?

Game autonomy GA1 (yes/no,REV) Is the game bound to specific places or local context?

GA2 (yes/no,REV) Does the game require configuring the physical space for a gamesession?

GA3 (yes/no,REV) Does the game require any kind of supervision (orchestration) whenplayers are playing it?

GA4 (yes/no,REV) Does the game require custom hardware, human actors, or other kind ofrelated resources?

Transmediality TM1 (yes/no,ELI) Does the game offer different participation modes (game roles)?

TM2 (yes/no,ELI) If the game offers different modes of participation, are these roles playedthrough different media (devices)?

TM3* (text&scale) Does the game balance the game experience for the various game roles?

TM4 (hint) Does the game need to use other media (e.g., desktop PCs) as support modules?

TM5 (hint) Is this game part of a broader transmedia experience?

Pers Ubiquit Comput

We chose to use the game itself in a custom evaluationprocess to assess it (IS8 information source). In this process,a researcher played the game on an Android device (MotorolaMoto X, 2nd Gen) for six days (one hour each day) in Rio deJaneiro, Brazil. In the first three days, the process consisted offree-play—the researcher played the game as a casual player.In the next three days, the researcher played the game whileobserving aspects about the pervasive qualities and takingnotes about them—this means playing the game while focus-ing on each question pertaining to the given quality at a time.On the last day, the researcher wrapped up the assessmentprocess with a final review of the written material generatedon the previous days. Compared to the general process wedescribed in Fig. 2, we summarize this hands-on process asfollows:

Step 1: Searching for available information sources—theresults include the game website, videos, interviews, reports,academic papers, and the game itself;

Step 2: We selected the game website for initial impres-sions about the game—we learned that Ingress is a pervasivemobile game (Step 3). We discarded this information sourceand selected the game application itself;

Step 4 and 5: Casual game playing, where the goal was toget acquainted with the game (i.e., the information source) andstart observing aspects related to the first-level qualities.

Step 6: Focused game playing, considering each second-level quality at a time and taking notes about the relevantaspects (i.e., using the checklists as a guide). At the end ofthis step, we filled out the quality assessment spreadsheet,reviewed the produced material, and wrapped up the process.

3.2.1 Accessibility

Considering Access usability (Usa1 = 0, Usa2 = 0), Ingressrequires players to focus extensively on the device screen(Usa1, Usa2). The game uses audio for output but it doesnot play a relevant role in player interaction. This assessmentanalysis encourages the designer to look after game activitiesthat require players to look less at the device screen and moreat the real world.

Ingress game has device independence (i.e., Device inde-pendence (DI1 = 1)) because it is possible to play Ingress inAndroid and iOS devices.

Regarding Game autonomy (GA1 = 0, GA2 = 1, GA3 = 1,GA4 = 1), Ingress is not bound to a specific game area butinstead depends on location-specific content—Bportals^ arereal landmarks (GA1). Game administrators create portalsusing software tools and players detect these portals throughGPS coordinates—therefore, it is not necessary to set up thephysical space to play (GA2). Ingress does not require orches-tration or supervision in live game play (GA3), but gameadministrators carry out some tasks periodically (e.g., creating

portals, banning misbehaving users). Finally, the game re-quires only off-the-shelf smartphones to play (GA4).

Considering Transmediality (TM1= 1, TM2= 0, TM3= 0,TM14 = −, TM5 = −), there are two roles in Ingress(BEnlightened^ and BResistance^)—therefore, the answer toquestion TM1 is Byes.^However, these two roles use the samegame interface (question TM2 is Bno^), which means thatTransmediality is totally unsupported (because TM2 is aneliminative question). Yet, question TM3 does not apply.Furthermore, Ingress does not use other media as supportmodules (TM4), and it is not part of a broader transmediaexperience (TM5).

3.2.2 Sociality

Regarding Social communication (SC1 = 1, SC2 = 0, SC3 =0, SC4 = 1, SC5 = 1), Ingress is a game that encourages teamplay (SC1). The game provides an internal chat interfacewhere players of the same team may start communicatingamong themselves. The game developers also use existingsocial networks (e.g., Facebook, Google+) to create commu-nities around the game, which means that players on the sameteam are able to discuss game tactics inside and outside thegame. There are live events announced in advance at the offi-cial game website (https://www.ingress.com/events), wherethe goal is to gather people to play the game together. Theseevents include a meet-up after the playing is over (SC1, SC4).For SC2, we were unable to determine whether Ingress trans-forms the relationship among players, but we believe that thegame may help in this issue due to its aforementioned aspects.As far as SC3 is concerned, players are not required to ap-proach strangers to start interaction. Lastly, Ingress presentsemergent gameplay (SC5)—for example, players are able tocreate several missions that are available to other players (theBmission game mode^).

Concerning Conformance to physical and social settings(CPS1 = 0.5, CPS2 = 0.75, CPS3 = 0.75, CPS4 = 0.75,CPS5 = 0.75), we evaluated question CPS1 (about player pri-vacy) as 0.5 because although Ingress identifies playersthrough nicknames, we do not know how this game handlesother player information (e.g., location). According to ourexperience, game activities in Ingress were not designed todisturb non-players nor embarrass players (CPS2, CPS3),and they seem to be fine with local conventions and etiquette(CPS4). Also, game activities in Ingress seem to be adequateto the physical setting where the game takes place (CPS5),which are mostly urban environments. Nevertheless, playersmight be discouraged from going after portals that are locatedin dangerous neighborhoods. However, conformance to phys-ical and social settings (CPS1 to CPS5) is a complex issue,because it depends on the player’s actual behavior whileplaying, which may be unpredictable. In fact, the media hasbeen reporting several safety and privacy problems with

Pers Ubiquit Comput

Ingress—such as players being caught by the police, beingrobbed in remote areas, and fatalities due to distraction.7

Therefore, we do not consider Ingress as fully supportingCPS. This quality assessment may encourage the game de-signer to think about important improvements. For instance,the designer may incorporate the need to pay attention to dan-gerous places (e.g., road crossings, cliffs, remote areas) as partof the gameplay, with rewards to the player.

Finally, Ingress does not support the quality Involving non-players (INP1 = 0, INP2 = 0, INP3 = 0, INP4 = 0).

4 Related work

As far as we know, there are no works in the literature thatpropose methods to assess pervasive qualities in games. Also,there are few works that discuss or present taxonomies ofpervasive qualities. We describe these works in this section.However, although we share some common concerns with theauthors of those works, we are pursuing a different researchdirection on a deeper assessment level.

Broll et al. [35] discussed technological challenges to im-plement pervasive games and presented some technologicalrequirements, such as Blocalization,^ Bcommunication,^Bcrossmedia augmentation,^ Bdevices,^ Bauthoring and gameengine,^ and Borchestration and surveillance.^ The main con-cern in Broll et al. [35] was to discuss the actual technologycomponents to implement these requirements and middlewaresupport. We were also motivated by technological(computing) issues to create our taxonomy but we did notfocus on how to implement the pervasive qualities.

Jegers [36] proposed a model of player enjoyment in per-vasive games that extends an existing playing enjoymentmodel for traditional games (the Bgame flow model^ [37]).In this model, Jegers [36] identified distinguishing Bpervasivefeatures^ (i.e., qualities), such as Bmobile/place-independentgameplay,^ Bsocial interaction between players,^ andBintegration of the physical and virtual worlds.^ Jegers fo-cused his work on Bplayer enjoyment,^ which is a kind of

Bemotional requirement^ [2]. On the other hand, we focusedon what makes a game pervasive—i.e., Bpervasiveness.^

Guo et al. [38] proposed a conceptual framework forpervasiveness in games containing four pervasive quali-t ies (referred as Bperspectives^) : Btemporali ty,^Bmobility,^ Bperceptibility,^ and Bsociality.^ These per-spectives have some properties they identified as impor-tant. Guo et al. [38] also proposed a simple metric toevaluate these perspectives and presented a scoring ofseveral pervasive games according to their metric. Thework by Guo et al. [38] is geared towards design stud-ies independent of technical implementation, whereas inour taxonomy, we are motivated by technological(computing) aspects.

Nevelsteen [39] analyzed several pervasive games andsummarized his findings as a Bcomponent feature set,^ whichcorresponds to nine aspects he found important to create per-vasive games: Bvirtual game world with world persistence,^Bshared data space(s) with data persistence,^ Bheterogeneousdevices and systems,^ Bcontext-awareness,^ Broles, groups,hierarchies, permissions,^ Bcurrent and historical game state,^Bgame master intervention,^ Breconfiguration, authoring andscripting in run-time,^ and Bbidirectional diegetic and non-diegetic communication.^ Nevelsteen [39] created his taxon-omy in a bottom-up fashion—i.e., he was concerned withgame engines, middleware, and game architectures. By ana-lyzing these elements, he identified his component feature set.On the other hand, we are concerned with how a pervasivequality would impact the game implementation (i.e., a top-down concern) and the use of technology in games. Our ap-proaches seem to complement each other, as we discuss

7 See https://www.oodaloop.com/osint/homelandsecurity/2014/04/03/google-ingress-game-players-subject-of-law-enforcement-fusion-center-alert/concerning homeland security, and http://www.irishtimes.com/news/ireland/irish-news/online-gamer-died-on-poolbeg-pier-capture-mission-inquest-hears-1.2645636 reporting a fatal case.

Table 19 BTechnological requirements^ from Broll et al. [35] (first row) mapped to our taxonomy

Localization Communication Crossmediaaugmentation

Devices Authoring andgame engine

Orchestration and surveillance

Context-awareness Connectivity(Communicability)

Transmediality(Accessibility)

Transmediality(Accessibility)

– Game autonomy (Accessibility)

Mobility(Spatiality)

Conformance to physical and socialsettings (Sociality)

Table 20 BPervasive characteristics^ from Jegers [36] (first row)mapped to our taxonomy

Mobile/place-independent gameplay

Social interactionbetween players

Integration of thephysical and virtualworlds

Mobility (Spatiality) Socialcommunication(Sociality)

Game contentadaptability(Context-awareness)

Game contentadaptability(Context-awareness)

Game autonomy(Accessibility)

Pers Ubiquit Comput

design and requirements issues and Nevelsteen’s work dis-cusses middleware issues.

We are not working with user experience (UX), as it isusual in HCI (human-computer interaction). In the light ofUX, quantitative and qualitative studies with a pervasive gamecan be found elsewhere [40].

Tables 19, 20, 21, and 22 describemappings of our taxonomyto the works we described in this section [35, 36, 38, 39]. Thesemappings arrange concepts that are somehow related. However,these mappings do not mean that the other taxonomies cover allaspects that we described for the qualities in [15].

5 Conclusions

A pervasive game is very different from traditional mobilegames because it brings the game experience out of a deviceand into the real world. Although this type of game has beenaround for some time (at least since 2001), its characterizationand design methods are scarce in the literature. Pervasivegames are no longer restricted to laboratories of game studies;they now require practical guidelines for better designs. Forexample, Ingress (and Pokémon Go, a more recent case) be-came a worldwide phenomenon, with thousands of peoplewalking in the streets with smartphones to interact with thephysical environment.

In this paper, we proposed a new method to assess perva-sive qualities in pervasive mobile games, in line with the con-text we mentioned above. This method generates a qualityreport, which consists of a quality spreadsheet (containingmetric values and comments) and a quality vector(representing the quality profile of the game in the form of abar chart). Also, we can compare quality vectors by calculat-ing similarity values for two basic situations: one is for thesame game but from different information sources, and theother is for different games.

Although our paper’s title emphasizes Bassessment,^the knowledge to assess a pervasive game is also avail-able to designers and developers interested in improvingexisting games or creating new games. An importantaspect of improving a game is to understand wherethe game lies concerning known and desired qualities(as the ones this paper provides). The main benefit ofour process will come from the encoded knowledge re-use by software developers, game developers, and gamedesigners.

The encoded knowledge we provide as taxonomies of per-vasive games qualities aims to improve the quality of the finalproduct. This encoded knowledge contains different qualitycategories—some qualities relate to software developmentand other relate to end users or players. For example,Sections 2.4.4 and 2.4.7 describe qualities related to user ex-perience (e.g., Access usability, Conformance to physical andsocial settings). The knowledge we provide in some cases isBmeta knowledge,^ which is knowledge that will help practi-tioners to get to the desired knowledge. This is exactly thepoint of providing questions (checklists), as in theGoal/Question/Metric (GQM) paradigm [41].

As far as we know, no other work in the literature hasproposed a method to assess pervasive qualities in games.

Table 22 Part of our taxonomy(first row) mapped to theBcomponent feature set^ inNevelsteen [39]

Persistency(Permanence)

Game autonomy(Accessibility)

Transmediality anddevice independence(Accessibility)

Context-awareness Socialcommunication(Sociality)

Virtual gameworld withworldpersistence

Roles, groups,hierarchies,permissions

Heterogeneousdevices andsystems

Context-awareness Bidirectionaldiegetic andnon-diegeticcommunication

Shared dataspace(s) withdatapersistence

Game masterintervention

Roles, groups,hierarchies,permissions

Current andhistoricalgame state

Reconfiguration,authoring andscripting inrun-time

Bidirectionaldiegetic andnon-diegeticcommunication

Table 21 TeMPS’s perspectives fromGuo et al. [38] (first row)mappedto our taxonomy

Temporality Mobility Perceptibility Sociality

Daily life interleaving(Permanence)

Mobility(Spatiality)

Context-awareness Sociality

Pers Ubiquit Comput

Indeed, we did not find a similar work that delves further intothe analysis of pervasiveness in games, provides checklists fora detailed assessment task, offers a systematic way to exploreseveral sources of information, and proposes a set of custom-ized metrics. There are very few works dedicated to explorepervasive qualities in games, as we described in Section 4.Although in this paper we focused on pervasive mobilegames, we believe that the lessons learned heremay contributeto a better practice of pervasive quality assessment in the de-sign of ubiquitous systems in general.

The application of our method to some well-known gamesmay shed more light on their pervasive characteristics andidentify ways to improve their quality. For example, the as-sessment of Conformance to physical and social settings(CPS) reveals important design issues for improving currentand future design of location-based games (see the summaryof the case studies in Section 3).

Our method relies on several types of information sourceswe identified in Section 0: Bpapers,^ Bofficial site,^ Bvideos,^Bnews,^ Bdev articles,^ Binterviews,^ Bpresentations,^Bhands-on,^ and Bmarketing.^ This is not a complete set ofpossible information sources and we understand that in prac-tice some of these sources may provide better information(i.e., more accurate, more complete, and/or more reliable) thanothers. The literature on knowledge elicitation usually as-sumes information sources as trustful. However, it is possiblethat an information source may not be reliable, and as such, itmay provide inaccurate or wrong information. A solution toavoid this problem is to rely on different information sources,which will provide different viewpoints and perspectives [42]on the context at hand. This solution may raise informationconflicts, and in these cases, conflict resolution [43] will be upto the engineer who is building the encoded knowledge.

We acknowledge the fact that the proposed method has ahigh cost in manpower and time. However, we claim that thiscost will be a wise investment to improve quality and reducerework. Moreover, we believe that future works may contrib-ute to reduce the cost by providing automated support for theproposed assessment process.

There are still many possible future research works. Forinstance, a more complete validation of our method shouldbe conducted by a large number of game designers. We aregranting public access to our spreadsheet template and internalreports8 to encourage this validation work. Moreover, de-signers of current pervasive games are invited to apply themethod and identify new ways to improve the quality of theirproducts. Also, we expect that discovering new qualities,checklist questions, and improving the current ones mightsoon be undertaken.

Another future work is investigating how aesthetic aspects(i.e., how the artifact is perceived) might impact pervasiveness

from the perspective of user experience in HCI (human-com-puter interaction). However, from the perspective of games,we would have the additional issue that usability and playabil-ity have distinct characteristics [44].

A future work of utmost importance is a more thoroughstudy on non-functional requirements related to safety [45],privacy, and health aspects of players and passersby [46]. Thenumerous reports of unexpected drawbacks about these issuesin recent pervasive games (e.g., Pokémon Go) reinforce theneed for this study. Lastly, we envisage the evolution of ourmethod towards other types of ubiquitous applications, inwhich abstract qualities are at a premium, such as being fun,immersive, empathetic, or affective.

Acknowledgements This research was supported by the followingBrazilian government agencies: CAPES (Federal Agency for Supportand Evaluation of Graduate Education, linked to the Ministry ofEducation); CNPq (National Council for Scientific and TechnologicalDevelopment) and FINEP (Brazilian Innovation Agency), which belongto the MTCIC (Ministry of Science, Technology, Innovation andCommunication); and FAPERJ (Research Support Foundation for theState of Rio de Janeiro). The authors are also thankful to NVIDIACorporation for the financial support.

References

1. Valente L, Feijó B, Ribeiro A, Clua E (2016) The concept of per-vasive virtuality and its application in digital entertainment systems.In: Wallner G, Kriglstein S, Hlavacs H, Malaka R, Lugmayr A, andYang H-S (eds) Entertainment computing - ICEC 2016. SpringerInternational Publishing, pp 187–198

2. Callele D, Neufeld E, Schneider K (2005) Requirements engineer-ing and the creative process in the video game industry. In: 13thIEEE International Conference on Requirements Engineering,2005. Proceedings, pp 240–250

3. Alves C, Ramalho G, Damasceno A (2007) Challenges in require-ments engineering for mobile games development: the meantimecase study. In: Requirements Engineering Conference, 2007. RE‘07. 15th IEEE International, pp 275–280

4. Furtado AWB, Santos ALM, Ramalho GL (2010) Streamliningdomain analysis for digital games product lines. In: Bosch J, LeeJ (eds) Software product lines: going beyond. Springer, BerlinHeidelberg, pp 316–330. https://doi.org/10.1007/978-3-642-15579-6_22

5. Bentley T, Johnston L, von Baggo K (2002) Putting some emotioninto requirements engineering. In: Proceedings of the 7th AustralianWorkshop on Requirements Engineering

6. Callele D, Neufeld E, Schneider K (2010) An introduction to expe-rience requirements. In: Requirements engineering conference(RE), 2010 18th IEEE international, pp 395–396

7. Björk S, Falk J, Hansson R, Ljungstrand P (2001) Pirates! Using thephysical world as a game board. In: Proceedings of Interact 2001.IOS Press, pp 9–13

8. Struppek M, Willis KS (2007) Botfighters. In: Borries F, Walz SP,and BöttgerM (eds) Space time play. Birkhäuser Basel, pp 226–227

9. Rashid O, Bamford W, Coulton P, Edwards R, Scheible J (2006)PAC-LAN: mixed-reality gaming with RFID-enabled mobilephones. Comput Entertain CIE 4:4

10. Ballagas R, Kratz SG, Borchers J, Yu E, Walz SP, Fuhr CO,Hovestadt L, Tann M (2007) REXplorer: a mobile, pervasive8 Please refer to http://www.ic.uff.br/~medialab/papers/2017/pauc

Pers Ubiquit Comput

spell-casting game for tourists. CHI 07 Ext. Abstr. Hum. FactorsComput Syst 1929–1934

11. S.T. Start: Zombies, Run!, https://zombiesrungame.com/12. Sra M, Schmandt C (2015) Expanding social mobile games beyond

the device screen. Pers Ubiquitous Comput 19(3-4):495–508.https://doi.org/10.1007/s00779-015-0845-0

13. Niantic Inc: Ingress, https://www.ingress.com/14. Niantic Inc: Pokémon Go, http://www.pokemongo.com/15. Valente L, Feijó B, Leite JCS (2015) do P.: mapping quality require-

ments for pervasive mobile games. Requir Eng 1–2916. Ryan MJ (2014) The role of stakeholders in requirements elicita-

tion. INCOSE Int Symp 24(1):16–26. https://doi.org/10.1002/j.2334-5837.2014.tb03131.x

17. Freeman P (1987) Software perspectives: the system is themessage.Addison-Wesley Longman Publishing Co., Inc., Boston

18. Boehm BW, Brown JR, LipowM (1976) Quantitative evaluation ofsoftware quality. In: Proceedings of the 2Nd InternationalConference on Software Engineering. IEEE Computer SocietyPress, Los Alamitos, CA, USA, pp 592–605

19. Boehm B, In H (1996) Identifying quality-requirement conflicts.IEEE Softw 13(2):25–35. https://doi.org/10.1109/52.506460

20. Fenton NE, Neil M (1999) A critique of software defect predictionmodels. IEEE Trans Softw Eng 25(5):675–689. https://doi.org/10.1109/32.815326

21. Fenton NE, Neil M (2000) Software metrics: roadmap. In:Proceedings of the Conference on The Future of SoftwareEngineering. ACM, New York, pp 357–370

22. Jiang Y, Cuki B, Menzies T, Bartlow N (2008) Comparing designand code metrics for software quality prediction. In: Proceedings ofthe 4th International Workshop on Predictor Models in SoftwareEngineering. ACM, New York, pp 11–18

23. Chung L, Nixon BA, Yu E, Mylopoulos J (1999) Non-functionalrequirements in software engineering. Spring

24. Leite JCSP, de Moraes EA, de Castro CEPS (2007) A strategy forinformation source identification. In: Anais doWER07 - Workshopem Engenharia de Requisitos, Toronto, pp 25–34

25. Valente L, Feijó B (2013) A survey on pervasive mobile games.Departamento de Informática, PUC-Rio, Rio de Janeiro

26. Lindt I, Ohlenburg J, Pankoke-Babatz U, Ghellal S (2007) A reporton the crossmedia game epidemic menace. Comput Entertain CIE5(1):8. https://doi.org/10.1145/1236224.1236237

27. Barkhuus L, Chalmers M, Tennent P, Hall M, Bell M, Sherwood S,Brown B (2005) Picking pockets on the lawn: the development oftactics and strategies in a mobile game. In: Beigl M, Intille S,Rekimoto J, Tokuda H (eds) UbiComp 2005: ubiquitous comput-ing. Springer, Berlin, p 903

28. Benford S, Anastasi R, Flintham M, Drozd A, Crabtree A,Greenhalgh C, Tandavanitj N, Adams M, Row-Farr J (2003)Coping with uncertainty in a location-based game

29. Bell M, Chalmers M, Barkhuus L, Hall M, Sherwood S, Tennent P,Brown B, Rowland D, Benford S, Capra M, Hampshire A (2006)Interweaving mobile games with everyday life. Proc SIGCHI ConfHum Factors Comput Syst:417–426

30. Benford S, Crabtree A, Flintham M, Drozd A, Anastasi R, PaxtonM, Tandavanitj N, Adams M, Row-Farr J (2006) Can you see me

now? ACM Trans Comput Hum Interact TOCHI 13(1):100–133.https://doi.org/10.1145/1143518.1143522

31. MontolaM, Stenros J,Wærn A (2009) Pervasive games: theory anddesign. Morgan Kaufmann

32. Good IJ (1977) Explicativity: a mathematical theory of explanationwith statistical applications. Proc R Soc Lond Ser Math Phys Sci354(1678):303–330. https://doi.org/10.1098/rspa.1977.0069

33. Chatzidimitris T, Gavalas D, Kasapakis V (2015) PacMap:Transferring PacMan to the physical realm. In: Giaffreda R,Vieriu R-L, Pasher E, Bendersky G, Jara AJ, Rodrigues JJPC,Dekel E, and Mandler B (eds) Internet of things. User-centric IoT.Springer International Publishing, pp 139–144

34. Flintham M, Greenhalgh C, Lodge T, Chamberlain A, Paxton M,Jacobs R, Watkins M, Shackford R (2011) A case study of explod-ing places, a mobile location-based game. In: Proceedings of the 8thInternational Conference on Advances in Computer EntertainmentTechnology. ACM, New York, NY, USA, pp 30:1–30:8

35. Broll W, Ohlenburg J, Lindt I, Herbst I, Braun A-K (2006) Meetingtechnology challenges of pervasive augmented reality games. In:Proceedings of 5th ACM SIGCOMM Workshop on Network andSystem Support for Games. ACM, New York, NY, USA

36. Jegers K (2007) Pervasive game flow: understanding player enjoy-ment in pervasive gaming. Comput Entertain CIE 5

37. Sweetser P, Wyeth P (2005) GameFlow: a model for evaluatingplayer enjoyment in games. Comput Entertain 3(3):3–3. https://doi.org/10.1145/1077246.1077253

38. Guo H, Trætteberg H, Wang AI, Zhu M (2010) TeMPS: a concep-tual framework for pervasive and social games. In: Digital Gameand Intelligent Toy Enhanced Learning, IEEE InternationalWorkshop on. IEEE Computer Society, Los Alamitos, CA, USA,pp 31–37

39. Nevelsteen KJL (2015) A survey of characteristic engine featuresfor technology-sustained pervasive games. Springer InternationalPublishing, Cham. https://doi.org/10.1007/978-3-319-17632-1

40. Blythe M, Reid J, Wright P, Geelhoed E (2006) Interdisciplinarycriticism: analysing the experience of riot! a location-sensitive dig-ital narrative. Behav Inf Technol 25:127–139

41. Basili V (1992) Software modeling and measurement: thegoal/question/metric paradigm. University of Maryland

42. Leite JCSP, Freeman PA (1991) Requirements validation throughviewpoint resolution. IEEE Trans Softw Eng 17(12):1253–1269.https://doi.org/10.1109/32.106986

43. Easterbrrok S, Nuseibeh B (1996) Using ViewPoints for inconsis-tency management. Softw Eng J 11(1):31–43. https://doi.org/10.1049/sej.1996.0004

44. Sánchez JLG, Vela FLG, Simarro FM, Padilla-Zea N (2012)Playability: analysing user experience in video games. Behav InfTechnol 31:1033–1054

45. Leveson NG (2012) Engineering a safer world: systems thinkingapplied to safety. The MIT Press, Cambridge

46. Chappell D, Eatough V, Davies MNO, Griffiths M (2006)EverQuest—it’s just a computer game right? An InterpretativePhenomenological Analysis Online Gaming Addiction. Int JMent Health Addict 4:205–216

Pers Ubiquit Comput