Andreas FlorosDept. of Audiovisual Arts, Ionian

University

Andreas FlorosDept. of Audiovisual Arts, Ionian

University

Nicolas - Alexander TatlasDept. of Electronic Engineering,

Technological Educational Institute of Piraeus

Nicolas - Alexander TatlasDept. of Electronic Engineering,

Technological Educational Institute of Piraeus

Stylianos PotirakisDept. of Electronic Engineering,

Technological Educational Institute of Piraeus

Stylianos PotirakisDept. of Electronic Engineering,

Technological Educational Institute of Piraeus

Sonic Perceptual Crossings: A tic-tac-toe Audio Game

Sonic Perceptual Crossings: A tic-tac-toe Audio Game

AudioMostly’11, September 7–9, 2011, Coimbra, PortugalAudioMostly’11, September 7–9, 2011, Coimbra, Portugal

Introduction

• Video - game concept:

- The visual component is critical for the game-play

- Audio information is used (mainly) for the ambient environment synthesis

• Audio - games

- Increasing research interest

- Sound represents a prominent component for realizing the required human-machine interaction interfaces

- Suitable for specific user target groups, such as visually impaired people

Introduction (cont’d)

• A definition attempt:

- Audio-games are game applications that employ appropriately synthesized auditory displays for developing the game-play scenario and establishing the user-computer interaction

• Sonic design is a key aspect for audio-games development

- Design perceptually efficient auditory interfaces

• Sonification

- Sonic design techniques for data (and game-world for the purposes of the current work) representation

Introduction (cont’d)

• The large variety of sonification techniques has allowed the development of multiple audio game genres

• We hereby discriminate the following two categories:

- a) audio games evolved based on existing (video) game scenarios

‣ scenario adaptation may be required, taking into account the narrative distinctiveness of an auditory environment

- b) audio game scenarios developed from the scratch, targeted to be realized using auditory displays only

Motivation

• This work focuses on the design / realization of the tic-tac-toe game

- We use a novel auditory and gestural interface combination

- Suitable for execution in mobile devices and platforms

• We attempt to focus on sonic design issues:

- investigate the performance of an advanced earcons scheme

- optimize playability performance

• Our approach incorporates a gestural / movement-tracking user interface that handles all the user movements and allows a completely eye-free game implementation

Grid-based games paricularities

• Wide acceptance: many legacy titles exist

- The players are already aware of the applied game rules

- Eliminate any requirements for describing the game-scenario details through audio-means only

• The game-play can be algorithmically described

- allow the employment of a sound design mechanism that takes into account the deterministic game rules

Applied sonification approach

• Employed sonification strategy: Earcons

- sounds synthesized by combining “fundamental” building sonic motives created using variable sound parameter values

‣ i.e. rhythm, pitch, timbre e.tc.

• Earcons pros and cons

- (+) Their synthesis approach allows the representation of concurrent (parallel) earcons, provided that specific design rules will be applied

‣ Concurrency can be more efficient provided that spatial characteristics in three dimensions (3D) are incorporated

- (-) Since they do not relate to their referent information in terms of the targeted context, user training is required to render them recognizable

Applied sonification approach: Design layer

• Earcons for “X” and “O”

- xij and oij

• Earcons for “X” and “O” placement

- xsij and osij

• empty cell earcon

- emptyij

• i and j are the grid coordinate indices

• boundary0X

- the rolling-sto wall auditory icon (will be explained later)

Applied sonification approach: Design layer details

• xij and oij

- constructed as simple note motives with variable pitch and rhythm parameters, using the following two rules:

‣ For consecutively increasing j-index values, the fundamental pitch frequency is doubled

‣ The rhythm of the overall note structure is proportional to the i-index values

• xsij and osij

- constructed by mixing the original xij and oij earcons with a very short impulsive click sound

• emptyij

- constructed as a smoothed, low intensity and very short duration “tapping” sound (the same for all cell-positions)

Applied sonification approach: spatial layer

• All earcons were spatially processed using binaural technology

Applied human control interface

• Basic requirement: eye-free control

• The auditory rolling pellet:

- assume the game grid as a two-axis revolving flat surface, with a pellet rolling on the top of it

- a rotation of the grid surface would force the pellet to roll towards the direction of the specific rotation

- the user can control the pellet instantaneous position by simply arranging the two rotation angles

‣ the rolling velocity of the pellet is considered constant and independent from the rotation angle in both control axes

Applied human control interface (cont’d)

• The auditory rolling pellet is also attached to an earcon depending on

- the spatial position of the cell grid it stands on

- the type of the cell: filled or non-filled

• It is reproduced only once (by the moment the pellet enters the specific cell)

- it is replaced by another earcon, only when the pellet enters a different cell

Applied human control interface (cont’d)

• Pellet motion tracking

- based on the data provided by a 2-axis gyroscope

• “X” or “O” grid-cell fill

- the user rapidly shakes the control equipment (an accelerometer is required)

• Grid boundaries determination:

- When the auditory rolling pellet reaches the grid boundaries, it “crashes” on a virtual wall causing it to stop rolling

- A corresponding spatial roll-stop auditory icon is then reproduced (boundary0X)

Game architecture (software/hardware)

Results - Test methodology overview

• Audio-game design and game-play assesment

• A two-level subjective test was employed

- Level #1:

‣ consider the sonic design efficiency, taking into account the particular requirements imposed by the game scenario and rules

- Level #2:

‣ play the game in real-world conditions

Results - Level #1 tests

• 20 non-audio expert adult subjects participated

• All subjects were first given a demonstration of the system together with:

- a detailed explanation of the correlation of the different earcons to the normal visual application state

- an analytic description of the interaction / navigation possibilities

• Two software applications were developed and utilized

- one for measuring the spatialization accuracy for each set of earcons

- one for assessing the user ability to imprint a given grid state from a specific auditory display state

Results - Level #1 test applications

• Measuring the spatialization accuracy

- buttons for playing back the earcons sounds on demand and proceeding to the next sample,

- nine buttons corresponding to the tic-tac-toe nine grid-cells positions

- the users were prompted to select the perceived position for each sample before proceeding to the next one

Results - Level #1 test applications (cont’d)

• Investigating the user ability to imprint a given grid state

- A significant “metric” associated with the playability of the game in terms of the earcons’ design followed

- Users were requested to select a state, namely “X”, “O” or “empty” for each grid position following the scenario played back

- 5 testing scenarios:

‣ 1 - 3 mainly for occupied positions (“X” or “O”)

‣ 4 -5 mainly for the “empty” earcons

Results - Level #1 tests outcome

Average correct placement for the earcons employed, nine positions per set

Average correct placement for the five test scenarios

Results - Level #1 tests outcome (cont’d)

• Less than 30% of the “empty” earcons are correctly placed

- however, more than 70% of the “O” and “X” cues are accurately positioned, indicating the strong association between the actual earcon design and their spatial perception

• Scenarios 1 and 2 results differ minimally compared to the results for scenarios 4 and 5, respectively

- the presence of “empty” earcons does not seem to affect the subject mapping capability

Results - Level #1 tests outcome (cont’d)

• Simple auditory display scenarios, including a small number of occupied blocks are easily mapped, with an accuracy percentage up to 80%.

• The minimum percentage of accuracy for

- the auditory display mapping on a scenario basis is 45%

- on a grid block basis the minimum is 75%

- Thus, a limited number of erroneous choices lead to mapped scenarios being dismissed

Results - Level #2 tests

• the users were allowed to play multiple tic-tac-toe sessions within a limited time interval.

• At the beginning this was a relatively difficult task

- it mainly resulted into random user actions and symbol placements on the tic-ta-toe grid area

• It finally turned out that after a maximum of three repetitions, the game-play was natural and feasible

• All subjects responded positively to the question regarding the ease of game-control through the auditory rolling pellet mechanism

• They also verified that the proposed earcons sonic design conceptually fits to the employed navigation mechanism

Conclusions

• We demonstrate and assess a tic-tac-toe game adaption to an audio-only environment

• Implementation suitable to be supported by any mobile platform equipped by specific movement-tracking accelerometer and gyroscopic sensors

• Earcons are employed as the fundamental means of sonifying the information required to construct the necessary auditory display

Conclusions (cont’d)

• Earcons’ design overview

- was an iterative process, leading to a robust and efficient set of spatialized earcons.

- although earcons concurrent / parallel representation was not required, however, the spatial characteristics of the sonic motives enhanced the degree of user immersion.

- the final sonic design considered the user control mechanism developed and employed, providing an integrated, multimodal interface for playing the game

Conclusions (cont’d)

• The efficiency of the audio-game prototype was assessed following a sequence of subjective tests

• Most subjects pointed out that

- considerable effort was required to confirm the possible position selection through the relevant audio playback, a fact also confirmed by the assessment results

- considerable skill was necessary in order to visualize the game state and provide the next input choice; obviously, this “complexity” increases through each game step

- while the discrimination between earcons for “X” and “O” was apparent, some found it difficult to specifically identify the “X” and “O” symbols during the testing phase

- the game play is natural and enjoyable, requiring though focused attention

Future work

• Further playability testing

- track intended and actual game inputs

- correlate the time required for visual and audio game play, as a metric of the possible effort required

• Tests by visually impaired subjects are expected to substantially differentiate the assessment results, given their increased abilities to visualize the game grid and actions

• User centric adaptation: allow users to define their own earcons mappings from a pre-defined, limited set